Ask The Performance Team articles

Collecting Debug Information from Containerized Applications

Becky — Wed, 17 Jan 2024 04:56:51 GMT

Collecting Debug Information from Containerized Applications

1. General Info

Howdy everyone! It's your favorite Debug Engineer, Will Aftring back again, this time talking about containers. This blog post will assume that you have a fundamental understanding of Windows containers. If that isn't the case, then then I highly recommend reading Get started: Run your first Windows container.

Many developers and IT Admins are in the midst of migrating long standing applications into containers to take advantage of the myriad of benefits made available with containerization.

NOTE: Not all applications are able to equally take advantage of the benefits of containerization. It is another tool for the toolbox to be used at your discretion.

But moving an existing application into a container can be a bit tricky. With this blog post I hope to help make that process a little bit easier for you.

2. Containerization steps:

2.1.1. Identify your dependencies

One of the benefits of containers is the limited deployment size that can be leveraged by Windows containers. A Windows Server Nano container can be as small as a few hundred megabytes!

However, as a part of trimming down the image size for the base container image, many things have been removed.

For example, in a Windows Server Nano container, much of the .NET framework has been removed. Meaning if your application is dependent on the .NET framework, then using a nano container image isn't viable.

Now application dependencies can be both within the Operating System (OS) or from the application itself. If you have segmented your application into executables and Dynamic Link Libraries (DLLs) then the container image needs to have those files available within the container image.

2.2.2. Find your configurations

In the Windows world, many developers manage the storage of configuration for their application via the Windows Registry. As a part of the containerization of the application, it will no longer share a registry hive with the container host. If you want your applications registry keys in place then you must add them either in the deployment of the container image or in the runtime of the application itself.

This also extends to local files and environment variables.

2.3.3. Figuring out what you need to communicate with over the network

As Ned Pyle so elegantly put it in Accelerating Your IT Career.

It's hard to find an IT system talking only to itself. Notepad, maybe (until you save a file to a network share).

It is almost guaranteed that your application will be leveraging the network at least to some degree. If it is a device on another container host then your problems are relatively simple:

How are you going to resolve the name of the host running the relevant network endpoint?
How is that outbound communication going to make its way back to the container?
- If you are using a NAT or Transparent network driver, then the behavior is similar to that of a VM
- If you are trying to use an L2Bridge then you need to make sure that the device you are communicating with is within the same LAN

If you are intending on containerizing the endpoint workload as well then things get a bit more complex. Intra-container communication is outside the scope of this blog post but if you are reading my other series on Introduction to Network Trace Analysis, then you should have some good fundamentals in place to understand this communication.

3. I've moved my application into a container and it isn't working. Help me!

First, don't panic. If application configuration were easy, it would be called baseball.

There are a few things we can do to get started with understanding the issue.

3.1.1. Is there anything logged to the console?

Anything that is written to the standard output (stdout) stream or standard error (stderr) is accessible from the container host.

Depending on your container runtime interface (CRI) you can use one of the following commands to read this output.

With docker
- docker logs <container id>
With Containerd
- ctr task attach <container id>

3.2.2. Are there any relevant log files being written to the disk?

Collecting log files is often an easier option. As a part of the container configuration, you are able to create a mapping between directories on the container host and directories within the container.

For the sake of being CRI independent you can use the Windows tool hcsdiag to make the location.

For example, if you are writing your log files to C:\Logs in the container, you can map that location to C:\<ServiceName>\Logs on the container host and read them in real-time.

hcsdiag share <container id> C:\Demo\Logs C:\Logs

Then read the log files from C:\Demo\Logs on the container host.

3.3.3. Leveraging External Tools

In (most) container images, many of your typical Windows troubleshooting tools are still available. And through hcsdiag, you can invoke commands directly in the container.

You can use wevtutil to export event logs.
You can use klist to display kerberos tickets
You can use Windows Error Reporting (WER) to collect process crash dumps

And through the mapping of a container directory to a container host directory, you can easily access the relevant debugging data.

Combining a mapped directory with the ability to run commands from within the container you can run all your favorite debugging tools from within the container.

For example, if you wanted to use procdump as the post-mortem debugger, you can do so:

Identify the container id
1. hcsdiag list
Ensure that you have shared mapping between the container and the container host
1. hcsdiag share <container id> C:\Dumps C:\Dumps
2. Please note, this mapping will remain until a restart of the container host
Setup procdump as the post-mortem debugger
1. hcsdiag exec <container id> "C:\Dumps\procdump.exe" "-accepteula" "-i" "-ma"
Wait for the crash to occur and it will be written to C:\Dumps on the container host

And using the basic format, substitute in your tool of choice for what you would like to do.

3.4.4. Memory Dumps

We all know our applications don't live in isolation. They need to read and write files and rely upon the Windows kernel. In some cases when debugging something like this you may want to collect a memory dump of the machine.

The good news is if you are running your containers in process isolation mode (the default) all you need to do is collect a memory dump of the container host.

Since the container host and the container share the same Windows Kernel, if you collect a complete memory dump of the container host, then the processes running within the container will be captured as well.

Within the memory dump the items in the container will present themselves as if they are running in a different session but you can proceed debugging to your hearts content.

4. Summarizing

See? Not so bad! Once you learn a few tips and tricks it isn't so different than debugging and application running anywhere else in Windows. I hope you find this post helpful in expediting your lift and shift projects.

Happy debugging!

How to Use TSS to Collect Data and Analyze to Solve High CPU Issues

Becky — Mon, 26 Feb 2024 17:08:51 GMT

How to Use TSS to Collect Data and Analyze to Solve High CPU Issues.

Hello everyone, this is Denzel with the Windows Performance Team. I found a tool that actively collects different data based on scenarios and streamlines the data collection process. Drumroll – introducing TSS (Troubleshooting Support Script). In my job, I see a lot of High CPU cases and collecting an ETL trace using TSS with Xperf aka WPR for high CPU has been fundamental in resolving issues.

I’d like to share some instructions, methods, and just insight on the tools in general that should be able to empower IT professionals resolve issues. This post will show how the TSS tool can work with the Windows Performance Recorder. TSS also works with several tools as it is very powerful but will focus on collecting a WPR trace using TSS when regarding a case of High CPU. I can even give you a great clue as to how to collect data for Intermittent High CPU cases as well! Once you have the data, I’ll then show you how to analyze it. Lastly, I’ll provide some additional resources on WPA Analysis for High CPU.

Data Collection Tools:

TSS

TSS (TroubleShootingScript) is a code signed, PowerShell based Tool and Framework for rapid flexible data collection with a goal to resolve customer support cases in the most efficient and secure way. TSS offers an extensible framework for developers and engineers to incorporate their specific tracing scenarios.

WPR/Xperf

“Windows Performance Recorder (WPR) is a performance recording tool that is based on Event Tracing for Windows (ETW). It records system and application events that you can then analyze by using Windows Performance Analyzer (WPA). You can use WPR together with Windows Performance Analyzer (WPA) to investigate particular areas of performance and to gain an overall understanding of resource consumption.”

*Xperf is strictly a command line tool, and it can be used interchangeably with the WPR tool.*

_________________________________________________________________________________________________________________________________________________

Let’s Dig in!

You notice your server or device is running with 90% high CPU. Your users are complaining of latency and poor performance. You have checked task manager, resource monitor or even downloaded and opened process explorer but there is still no exact root resource glaring you in the face. No worries, a WPR will break down the high CPU processes a bit more. You could even skip straight to this step in the future when you get comfortable working with these tools.

Setup TSS

Running a TSS troubleshooting script with the parameters for either WPR or Xperf gathers some granular Performance data on machines showing the issue. In the example below, I’m saving the TSS the script to 😧 (note the default data location is c:\MS_Data). In your web browser, download TSS.zip found here: https://aka.ms/getTSS or you can Open an Administrative PowerShell Prompt and paste the following commands.

The commands below will automatically prepare the machine to run TSS by taking the following actions in the given order:

Create D:\TSS folder
Set the PowerShell script execution policy to RemoteSigned for the Process level (process level changes only affect the current PowerShell window)
Set TLS type to 1.2 and download the TSS zip file from Microsoft
Expand the TSS.zip file into D:\TSS folder
Change to D:\TSS folder

Ex: Commands used below

md D:\TSS

Set-ExecutionPolicy -scope Process -ExecutionPolicy RemoteSigned -Force

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Start-BitsTransfer https://aka.ms/getTSS -Destination D:\TSS\TSS.zip

Expand-Archive -LiteralPath D:\TSS\TSS.zip -DestinationPath D:\TSS -force

cd D:\TSS

The result will be a folder named TSS on drive D.

_________________________________________________________________________________________________________________________________________________

Gathering Data using TSS

Open an elevated PowerShell window (or start PowerShell with elevated privileges) and change the directory to this folder:

cd D:\TSS\

*WARNING* Data collection grows rather large quickly. You should have at least 30% of your overall RAM available as hard drive space. (Example, if you have 8 GB of RAM – then the file can grow to 2.5GB or larger in c:\MS_Data.)

What are some of the scenarios you might have? Maybe you want to manually collect the trace. Or, once you start the trace, let it automatically stop.

How about limiting the file size? There are several parameters you can adjust for your needs.

Below you will find variations of using TSS to collect WPR data in high CPU occurrences. You have an option of using either WPR or Xperf commands. Please review all of them before deciding which trace to take for your environment.

1. Scenario In State needing user intervention to stop the trace: The issue is currently occurring, and the following example needs user intervention to stop the trace. The WPR can grow to 80% of the memory with the example commands listed below.

.\Tss.ps1 -WPR General *** (run it for 60 seconds to no longer than 3 minutes)
.\Tss.ps1 -Xperf CPU ***(run it for 60 seconds to no longer than 3 minutes)

Default location of saved data will be C:\MS_Data.

The prompt will tell you when to reproduce the issue, just simply enter “Y” will END the trace at that time and the machine in question experiencing high CPU will then finish running the data collection.

2. Scenario In State but you wanted to limit the size and length of time: The issue is currently occurring; the following example does NOT need user intervention to stop the trace. Default location of saved data will be C:\MS_Data. The Xperf can grow to 4GB of memory and runs for 5 minutes with the setting below:

.\TSS.ps1 -Xperf CPU -XperfMaxFileMB 4096 -StopWaitTimeInSec 300

Note: you can modify the size and length of the trace by increasing or decreasing -XperfMaxFileMB and -StopWaitTimeInSec when it is initially run.

3. Scenario In State but you wanted to limit the size and length of time with data saved on Z:\Data drive instead of C:\MS_DATA (default): The issue is currently occurring; the following example does NOT need user intervention to stop the trace. The Xperf can grow to 4GB of the memory and runs for 5 minutes with the setting below and this time the resulting data will be saved on Z:\Data. You simply need to add -LogFolderPath Z:\Data to the command.

.\TSS.ps1 -Xperf CPU -XperfMaxFileMB 4096 -StopWaitTimeInSec 300 -LogFolderPath Z:\Data

4. Scenario Intermittent High CPU and having a tough time capturing data: These commands will wait for the CPU to reach 90%, start a trace and will stop the file from growing larger than 4GB while running for 5 minutes.

.\TSS.ps1 -Xperf CPU -WaitEvent HighCPU:90 -XperfMaxFileMB 4096 -StopWaitTimeInSec 300

5. Scenario Intermittent High CPU and having a tough time capturing data: These commands will wait for the CPU to reach 90%, start a trace and will stop the file from growing larger than 4GB while running for 100 seconds (1.5 minutes roughly).

.\TSS.ps1 -Xperf CPU -WaitEvent HighCPU:90 -StopWaitTimeInSec 100

Pro Tip: You can check for additional Xperf/WPR commands by doing a search on the help command files in TSS by typing

.\TSS.ps1 -help at the prompt. When prompted to enter a number or keyword, type xperf or wpr, hit enter, and you will see the options.

Ex: Finding help with keyword ‘xperf’

Be sure to wait for the TSS script to finish, it can take some time (even an hour to finish writing out). The PowerShell will return to type line and the folder in C:\MS_Data should zip itself when complete. The location of the script does not determine the location of the data collected. Wait for trace to finish before exiting PowerShell.

Reminder: Just like in the first trace, you learned data collection grows rather large quickly. You should have at least 30% of your overall RAM available as hard drive space. (Example, if you have 8 GB of RAM – then the file can grow to 2.5GB or larger on c:\MS_Data.)

_________________________________________________________________________________________________________________________________________________

You have the Data – Now Let’s look at it!

Download the Windows ADK (Windows Assessment and Deployment Kit) from this location: Download and install the Windows ADK | Microsoft Learn. Once you download the Windows ADK, you want to install the Windows Performance Toolkit. Double click on the executable (.exe) to start the installation process.

Uncheck everything except Windows Performance Toolkit, then click Install.

Opening the data in the C:\MS_DATA folder

When complete, the WPR general TSS command should have placed all collected data into this folder in a zipped file. You will know the trace ran all the way without stopping prematurely when you see the zipped file in C:\MS_DATA. There will also be a message in the PowerShell window when the diagnostic completes stating the name of and the location of the zipped file.

You will need to unzip the zipped file to analyze the WPR trace (.etl file). After unzipping, you will see several data collections that can be helpful with analysis. However, what you mainly want to look at is the .etl file which is usually the biggest file located in the folder.

If you double click the .ETL file it should open in WPA, but if not, you can manually open the newly installed application and navigate to your file.

Example:

You can open the .ETL file to view the WPR trace with WPA (Windows Performance Analyzer) by clicking File, Open and then browsing to the file that ends with the .ETL extension.

Step 1. Open WPR trace in WPA and load the Public Symbols. You may also see symbols listed from the NGEN folder (NGEN is part of the folder name) collected at the time the WPR trace was run.

Select Trace, select Configure Symbol Paths

Click + sign (highlighted in yellow in screenshot below), then enter Public Symbol Path: srv*c:\symbols*https://msdl.microsoft.com/download/symbols

More Information: (Symbol path for Windows debuggers - Windows drivers | Microsoft Learn)

Once symbols are configured simply click Load Symbols

Step 2. Once open you should see a window similar to the screenshot below. Expand Computation on the left and drag CPU Usage (Sampled) to the right side of the Window to load. You can also double click CPU Usage (Sampled) for it appear on the right side.

You will then see space on the top graph showing, “Trace Rundown”. That is not needed as it is the part of the trace where the script was finishing up. To get rid of the trace rundown, highlight the area before trace rundown, right click, then select “Zoom”.

You can now filter down each of your processes deeper and deeper to try to locate a potential root cause of what is spiking the CPU. You can look to see which processes have the highest weight over on the right-hand columns to help pinpoint the highest consumers. It may be a specific kernel driver, application, process, etc. but this should help point you in the right direction of what process is exhausting resources.

These are the columns you will want to focus on:

Left of Gold Bar:

Process

Stack

Right of Gold Bar:

Count

Weight

% Weight

You can see the CPU usage is the highest due to CPUSTRESS.EXE in this example. As you filter down you can see the threads that contribute to the max CPU spike which is visible in % weight sum. This can be helpful to find out which threads, functions and modules are called for the root cause.

Step 3. You can open graph, "Utilization By CPU" for additional analysis.

By adding the process column on the left side of the gold bar you can confirm which processes are the highest consumer % weight wise on each CPU. To add the process column, right click on one of the column titles and select more columns as instructed in screenshot below. You can then filter down each individual processor and view the CPU percentages per process.

Conclusion:

Once again this is not the only use for the TSS tool. But as you can see, the WPR/Xperf trace is a very complex tool that gathers data from a simple PowerShell command. This can be very efficient for troubleshooting. This article is not meant to cover all scenarios. However, I highly recommend taking some time to learn more about what TSS can accomplish as this tool will only continue to get better.

If at any point you get stuck don’t hesitate to open a support case with Microsoft.

Additional Information:

Information on TSS and alternative download site:

Introduction to TroubleShootingScript toolset (TSS) - Windows Client | Microsoft Learn

Information about Windows Performance Toolkit

Windows Performance Toolkit | Microsoft Learn

For Reference:

Download Windows Assessment Toolkit which contains Windows Performance Analyzer

Download and install the Windows ADK | Microsoft Learn

How to setup public symbols

Symbol path for Windows debuggers - Windows drivers | Microsoft Learn

Understanding Lock Contention in Windows Performance Analyzer (WPA)

Becky — Wed, 17 Jan 2024 04:53:22 GMT

Understanding Lock Contention in WPA

Hi everyone, this is Will Aftring with the Windows Debug team. I was debugging an application performance issue and thought “this is a great example of lock contention”.

For this post we will be using the Windows Performance Analyzer (WPA) to review data collected with the Windows Performance Recorder (WPR). For the sake of keeping this post focused I won’t go in depth on WPR but there are plenty of resources on how to get started. Getting Started Windows Performance Recorder | Microsoft Docs

Let’s start with some vocabulary regarding thread states.

Schedule a thread: To schedule a thread is to ensure that it is the next thread to run on the CPU
Preempt a thread: To remove a thread from a processor before it has completed its work or yielded for another thread to run.
Ready: This thread is ready to run and is consideration for next to hop on the CPU
Deferred: This thread is waiting to run on a specific processor but hasn’t been scheduled yet.
Standby: This thread has been selected to run next on a specific processor. When a specific condition is met, the dispatcher switches in the thread.
- Important: Only one thread can be in standby state for each processor on the system
Running: This thread is currently on the CPU doing, performing work
Waiting: A thread that is waiting for a specific condition to be met. This can be either voluntary (ie WaitForSingleObject) or involuntary (waiting for memory to be paged-in). When the condition is met the thread is moved back into a ready state.
- It is important to keep in mind that Windows does not follow a FIFO model for waiting threads.
Transition: A thread enters this state if it’s ready for execution, but its kernel stack is paged out of memory. Once its kernel stack is brough back into memory, the thread enters a ready state.
Terminated: When a thread finishes executing or is told to terminate (TerminateThread), it enters this state.

Here is a flow diagram that covers the thread interaction:

Now that we have laid our groundwork for thread states, let’s jump into some analysis.

Below I have the CPU precise view open.

Let’s clarify what these columns mean.

Name	Description
New Process	The process being readied
New Thread Id	The thread id within the process specified in New Process
New Thread Stack	The call stack that is running during a specific time
Ready Thread Stack	The call stack that readied the new thread stack
Readying Process	The process owning the ready thread stack
Readying Thread Id	Thread that owns the ready thread stack
Yellow Bar	This indicates a pivot point in the data and moving columns across this bar will change how the information is displayed
Count	Number of calls to this function
CPU Usage (sum)	The cumulative time the specified thread / process spends before it was switched back out
Ready time (sum)	The cumulative time the specified thread / process spent in readied state
Waits time (sum)	The cumulative time the specified thread / process spent in a waiting state
Ready time (max)	The maximum single ready time seen for the specified thread / process
Wait time (max)	The maximum single wait time seen for the specified thread / process

Understanding waits with an example program

I know I just threw two pages of definitions at you so let’s put this new info into practice with an example.

This program does the following:

main
- Initializes a Critical Section Object
- Creates 200 threads that will all run the CritThread function
- Wait for user input
CritThread
- Attempt to acquire a critical section
- Print to console
- Leave critical section

While any of the threads running the CritThread function owns the critical section, none of the other threads in this application can enter that critical section.

If we have 5 threads (1,2,3,4,5) and thread 1 calls EnterCriticalSection on g_CS, threads 2-5 will wait in EnterCriticalSection until thread 1 has called LeaveCriticalSection. You can think about this like driving a car, only one person can drive at a time.

Now that we understand how this interaction is supposed to work, what does it look like in WPR?

As we follow the cumulative wait times column, we can see that starting in the highlighted frame below, we have a big drop off in the wait time.

Here is a zoom in of the same highlighted frame above.

We can see thread 5652 is waiting for a critical section, but what does that mean in the context of the columns?

Count 110: This function was called 110 times
CPU Usage 0.791: This function spent 0.791 ms before the thread was switched out
Ready time (sum) 852.600: This function spends a cumulative 852 microseconds in a ready state
Wait times (sum) 4,320,242: This function spent a cumulative 4,340,242 microseconds in a ready state
Ready time (max) 33.700: The longest amount of time that the function spent in a ready state was 33.700 microseconds
Wait times (max) 183,466: The longest single wait time this function had was 183,466.800 microseconds

Let’s go even deeper. If we look at the ntdll.dll!RtlEnterCriticalSection we can see that the thread stack can be expanded further to show the readying stack of the thread that readied 5652.

In the red box, we can see the call stack of the threads that readied thread 5652.

In the green box, we can see the thread id of each thread that readied thread 5652.

Meaning that each of those threads at one time had the red box call stack which readied the new thread.

Let’s shift the view to the columns on the right side of the yellow bar, there are a few important things to keep in mind.

Breaking down thread 484, in the purple box there are a total of 3 threads (red, blue, and green) that have readied thread 5652.

Looking at our columns, we can see that each of those instances has its own value for the columns. This makes sense because each of these single instances is an occurrence of the readying thread stack.

Thinking through the display, we can see that each of the wait times from the readying thread contributes to the cumulative wait time. Now from the wait times displayed, none of the individual waits are particularly long.

With none of the waits being particularly long, the cumulative effect is death by a thousand paper cuts. Resulting in lock contention.

Important! Readying thread vs lock owner

When considering readying threads and locks it can be understandable to think that just because a readying thread is releasing a lock that it has owned that lock for the full duration of the wait.

While that may be true when there are only two threads, that is not necessarily true when you have more threads.

In this scenario, thread 1 will always ready thread 2 and thread 2 will always ready thread 1. And if it is taking a long time to acquire the critical section it is because the other thread must be doing something that is taking a long time.

But as we increase the number of threads, the complexity increases along with the waiting times.

In this scenario thread 2 readied thread 4 but while thread 4 was waiting the critical section was owned by:

Thread 1
Thread 3
Thread 2

But only thread 2 readied thread 4. Tricky right?

But so what?

We have been able to identify that the waits we are seeing in our application are primarily from many short waits causing the cumulative waiting time for a thread to be large.

What can we do about this?

The good news is when dealing with lock contention we get to be creative with our solution and have lots of flexibility to alleviate the issue.

To be clear, the wrong solution is to throw more threads at the issue. Each thread that we add that contends for a lock under contention, the waits get exponentially worse.

Below is a breakdown of time spent waiting for a critical section, as the number of threads waiting for the critical section increases, the longest individual wait increases.

Threads	Longest individual wait (ms)
2	18
4	86
16	152
256	1658

Clear as day we can see that the more threads we throw at a lock under contention, the longer each individual wait will get.

Now let’s jump into what we can do.

Considering lock usage

How am I using this lock?

When considering lock usage, it is typically to protect a specific resource and perform some work on the protected resource.

But the longer a resource is protected, the higher the likelihood that other threads will wait for the release of that resource.

Let’s use the following scenarios as an example:

We lock the entire function that modifies the resource

We lock the operations that modify the resource

These may look very similar but in scenario 2, the developer is leveraging a tight scope on when the critical section is locked. This prevents any delays from WriteEventLog from preventing other threads from reading or writing to pData.

The shorter the amount of time a lock is held, the less likely other threads are going to wait for that lock.

Does this lock meet program needs?

In all discussions so far, we have been talking about locking resources using critical sections.

Typically, the usage of a lock is to allow inter-thread synchronization to protect a specific resource to ensure that only one thread is operating on that resource at a time. But the question becomes, does this specific interaction need to be mutually exclusive?

Let’s say I have 100 threads. 99 threads read the value of a variable and print it to the console and 1 thread changes that variable’s value.

Is there any harm in having the 99 reader threads access the variable at the same time and only have the resource locked when the writer thread is operating on it?

If not, then maybe a Slim Reader/Writer (SRW) Locks is the correct lock for your program.

A resource that is locked with a reader lock will not contend with any other reader threads.

A resource that is under a writer lock will contend with both reader and writer threads.

From the SRW documentation:

Reader threads read data from a shared resource whereas writer threads write data to a shared resource. When multiple threads are reading and writing using a shared resource, exclusive locks such as a critical section or mutex can become a bottleneck if the reader threads run continuously but write operations are rare.

It's easy when thinking about thread synchronization as a case where everything looks like a nail and a critical section is the hammer. But it is important to look at all the tools you have in your toolbelt before you start swinging.

Can I distribute this work?

It can be convenient when scaling an app to scale vertically and create one very big strong machine. But there are still limitations to what one single machine can do.

If the bottleneck isn’t the available resources (CPU utilization, available memory, etc.) but waiting for a single resource, then how big your machine is less relevant.

You can often find better performance not by throwing all the work at a single machine but allowing multiple machines to perform the work.

For example, what if an application heavily relies on one domain controller (DC)? After initial DC discovery, the result will be stored in the DCLocator cache, and all subsequent work will be sent to that DC. If that DC is experiencing bottlenecks due to lock contention, then why not send the work to another DC? Or even better, don’t wait for that DC to encounter issues before distributing the work. If the information is available from any DC, why not send the work to any DC?

Do I need to be doing this much work in the first place?

This can be a tough pill to swallow but it is important to keep it in mind.

The correct scaling solution isn’t to throw work at a machine until it falls over, but rather to find an appropriate performance balance between necessary system functionalities and application workloads.

When work is bottlenecking an app, it may be worth considering if that work is necessary.

This often comes in the form of:

Frequency (Do I need to query this information X times per Y?)
Size (Do I need to query all items in database Y?)

The idea of pulling back is usually the quickest way to alleviate the symptoms. This can take the form of:

Reduces the number of threads contending for a lock
Reduces time spent while holding a lock

Pulling back is particularly relevant when the lock contention is occurring outside of the application. There are necessary locks within the operating system that multiple applications will leverage. If your application throws unnecessary amounts of work at this lock, everyone else who relies on this lock will suffer.

Wrapping up

At the end of the day, lock contention is an issue that arises frequently, and its solution is often a part of a larger conversation.

I understand that the suggestions above are not easy, but they are important. Not only within your application but within a healthy operating system. Applications have internal locks, but they run on the operating system which has its own locks which the application will leverage directly or indirectly. Applications running on an operating system need to behave like good roommates.

Clean up after yourself
Don’t hog the TV
Don’t use all the hot water
Etc…

Otherwise, everyone is going to suffer as a result of one bad roommate.

If you enjoyed this post and would like to know more, feel free to check out my posts on the Microsoft CISTech community at https://aka.ms/WillAftring.

Automating Data Collection for Memory, CPU, and Disk issues using CLUE

Becky — Wed, 13 Oct 2021 20:02:25 GMT

Automating Data Collection for Memory, CPU, and Disk issues

My name is Susan Buchanan, and I am a Support Escalation Engineer with the Windows Performance Team at Microsoft. Does Performance Data Collection got you singing the Blues? This blog addresses how to troubleshoot High CPU and unaccounted memory usage or memory leak to include identifying and data collection using the CLUE tool written by Clint Huffman. A special thank you to Ron Stock for writing me a bad driver to use in demonstration.

I want to discuss how to perform data collection when an issue may be intermittent, troublesome to catch, ongoing, or even reproducible at will. For many years, data collection was manual, time consuming, and often labor some.

Can’t seem to capture the right data at the right time?
Intermittent issues? Reproducible issues?
Do customers leave data captures on too long, and the log sizes are not useful, or too large?
Need a foolproof way for end users to capture data that isn’t too complicated?

It's frustrating when you obtain a set of data only to realize you needed an additional dataset and must perform the data capture again. You might start with a RAMMAP and a performance monitor on the first data collection, move to a pool monitor or WPR/Xperf on another collection, and then might realize you also needed a tasklist or to find which drivers were associated with which pool tags you need another set of data collected.

Performance is Complicated! It really is.

Well, those days are gone! Thanks to CLUE which runs an ETW trace in the background until a threshold found in the config.xml is hit and then automagically collects data via Tasks in Task Scheduler! This gives you a more robust view of what has occurred since the data capture shows prior to the event, and then the event in the ETW trace without manual intervention.

CLUE was written by a Microsoft employee who has made this code open source and extensible. You can modify the config.xml to create your own scenarios, increase the data captures, and much more.

CLUE Scenarios/Thresholds

Automatically collects a counter log and ETW trace whenever the following conditions occur:

Single CPU over 90% for more than 3 seconds.
Free System PTEs of less than 50000 (kernel address space is low)
System Committed memory greater than 90% (indicates one or more memory leaks that are consuming the system resources)
Kernel Pool Paged virtual address space greater than 10% of system committed memory (indicates a driver leak in pool paged)
Kernel Pool NonPaged virtual address space greater than 10% of physical memory (indicates a driver leak in pool nonpaged)
Disk latency of greater than 25 ms for 3 or more seconds (includes high consuming processes, disk filter drivers, page file usage, and more)
High CPU by WMI (includes WMI tracing to identify the query causing it)
High thermal temperatures (traces CPU and power usage)
High battery drains greater than 20% of battery capacity within one hour (traces CPU, GPU, and power usage)
Has a User-initiated trace start optimized for application hangs. The user can initiate a wait analysis trace to determine why Windows, or an application is hanging.

CLUE Installation

Requires administrator rights to install. After installation, non-admin users can use the device normally and data collection will still occur even for user-initiated data collections.

For Windows 10, Windows Server 2016 and 2019
1. Download and setup scheduled Tasks via Microsoft Clue tool 1.1.6: Clue/Clue_1.1.6.zip at master · Clint Huffman/Clue · GitHub
2. Confirm you have 2-4 GB free on your c: drive
3. To install CLUE run setup.bat as administrator
4. Accept the defaults

Data location stored (default)

C:\ClueOutput

Extensibility

CLUE is also extensible allowing for more performance rules by modifying the config.xml without changing the binaries.

GIT:

https://aka.ms/cluetool

Following are 2 scenarios describing how CLUE can be useful in troubleshooting High CPU and High Memory.

What is causing high CPU on my server?

Scenario #1: Issue of intermittent high CPU on a Windows Server 2016 where it would jump to > 90%.

In task manager we can easily see that it is the CPU Stress application; but what if it wasn’t so obvious? What if we needed to drill further down by CPU, or by stack?

Of course, the first line of defense if the issue is occurring in real time is to pop open Task Manager to see what it shows. This is not always possible which is another reason why CLUE comes in handy.

We see from a quick look in Task Manager the CPU is over 90%

Task Manager shows both kernel and user mode in the graph. Kernel mode is in (darker blue/grey area)

But what if the issue wasn’t in state? Or, what if Task Manager wasn’t so helpful?

Luckily, CLUE was running, and we know from the ETW trace what CPU usage looked like prior to and during the occurrence.

From the C:\ClueOutput folder we see the following data captured due to the ProcessorTime > 90% and DiskLatency > 25ms hit.

When we expand the zipped file for Processor Time > 90%, we see the following data captured:

When we are looking for high CPU, we typically want to look at the following counters:

Component

Performance Aspect being Monitored

Counters to Monitor

Processor

Usage

Processor\ % Processor Time (all instances)

Processor\% DPC Time

Processor\% Interrupt Time

Processor\% Privileged Time

Processor\% User Time

Processor

Bottlenecks

Processor\% Processor Time (all instances)

Processor\% DPC Time

Processor\% Interrupt Time

Processor\% Privileged Time

Processor\% User Time

Processor\Interrupts/sec

Processor\DPC’s Queued /sec

System\Context switches /sec

System\System Calls/sec

System\Processor Queue Length (all instances)

Processor

Counter Name	Metric
Processor Queue Length (PQL)	Divide the PQL into the number of processors ~2+ sustained per processor and high CPU present: Check processes for high CPU consumption, also check Context Switching, % DPC Time, and % Interrupt Time
% Processor Time	0-50% healthy 50-80% monitor / warning Monitor 80-100% critical. System may appear sluggish.
% DPC Time	~% Processor Time > 85% and % DPC Time > ~15%: investigate if they are constantly above these levels, short spikes are ok. If only on 1 processor ~100% Processor Time and ~50%+ DPC Time:
% Interrupt Time	High CPU Interrupt Time – more than ~30% interrupt time (A high amount of % Interrupt Time in the processor could indicate a hardware or driver problem). Very high CPU Interrupt Time – more than ~50% interrupt time (A very high amount of % Interrupt Time in the processor could indicate a hardware or driver problem)

Wow – our processor time is significantly high for much of our 6-minute performance monitor.

Check out the Processor Queue Length which is > 10 on average. That’s not looking good here. So, let’s drill down into who is using the %Processor Time.

Processor Minimum Maximum Average

=========================================================================

% Processor Time : 4.688% | 100% | 81.605%

When we add counters for Processor/%ProcessorTime and look at the instances we quickly see CPUStres64.exe is the highest consumer. There are even gaps where the %ProcessorTime exceeded 100 percent and went to around 180% in the data.

% Processor Time Minimum Maximum Average

=========================================================================

CPUSTRES64 : 0% | 180.309% | 19.787%

From the CLUE ETL (Windows Performance Recorder trace), we see the highest CPU is 0. Ok, in my example I only have 1 CPU – but it’s good to check if the CPUs are being consumed equally.

In another example, where I have multiple processors and CPUStres64.exe is the culprit we see where each processor is hit fairly, equally by the process.

Where did all my Memory Go? And why did my Server become unresponsive?

Scenario #2: Windows server 2019 that inevitably runs us of memory due to a resource depletion, but we cannot figure out why?

Looking at task manager is leaving you without a "clue" as the top consumer is just 75 MB.

If we were to manually gather the data, it would look like this:

Obtain a perfmon to determine there is a memory issue. Oh, there is, but now I need more data.

I might look at the following counters.

Memory

Usage

Memory\ Available Bytes
Memory\ Cache Bytes
Memory\%Committed Bytes
Memory\Pages Input or Reads/s

Memory

Bottlenecks or leaks

Memory\ Pages/sec
Memory\ Page Inputs or Reads/sec

Memory\ Page Output or Write/sec
Memory\ Transition Faults/sec
Memory\ Pool Paged Bytes

Memory\ Pool Paged Resident Bytes
Memory\ Pool NonPaged Bytes

Although not specifically Memory object counters, the following are also useful for memory analysis:
Paging File\ % Usage object (all instances)
Cache\ Data Map Hits %

Memory - some metrics to keep in mind when you look at the data.

Counter Name	Metric
Available Bytes	Minimum value: ~700mb – monitor ~500 mb or less - critical
Pool Paged Bytes (PP)	Exhausted: Event 2020 logged in system log. Both 2019 and 2020 event log errors with a source of SRV are relatively common and indicate a depletion of non-paged or paged pool memory respectively.
Pool Nonpaged Bytes (NPP)	Exhausted: Event 2019 and 2004 logged in system log. NPP can exhaust and it causes more of a problem if high or pool tag is leaking resulting in low memory condition on the system. Check usage if available memory is low on the OS
Free System PTEs	Rarely if at all will a 64-bit machine run out of PTEs as normally available is in the tens of millions. Available PTEs value is few hundred thousand: monitor Available PTEs value is 10,000 or less: critical
Handle Count	Exchange, SQL and LSASS can have 100k + and be normal. Other software or processes high are worth investigating. Rule of thumb is 1500 – 2000 is a good place to being looking at those applications consuming a lot more than this number.
Thread Count	~500+ possible unexpected behavior: monitor ~2000+: warning
% Committed Bytes Used	~90% but may only be relative if low available memory present
Pages/sec	~1000+ could be start of running out of memory: monitor ~2500+ could be experiencing system wide delays: Check available memory ~5000+ most likely experiencing system wide delays: Check available memory and memory used by individual processes

The manual way which would be lengthier and more time consuming would be to:

Obtain a poolmon to determine what tag might be leaking. Okay, there is a leak, but now I need more data.
Use FindStr to try and identify what tags are being used by what files (unfortunately, we rebooted the box, and we had to start all over with the data captures).
Obtain Tasklist to determine the services in use
May need an ETL or even a dump after that.

Here’s CLUE to the rescue. CLUE was running a Perfmon trace and noticed that NonPaged Pool consumption was > 10% for the following tags: LEAK and MxGN. It then automagically collects data needed to help you isolate this further (like Perfmon, Poolmon, and in some instances an ETL trace).

Looks like my good friend, Ron wrote me a bad driver that was allocating but not deallocating my memory using the LEAK Tag.

*Normally I would show a larger time frame between poolmon data, but this is just for demo purposes. Clearly you see in a matter of minutes the number of Frees is 0 and the number of Bytes is growing.

LeakyFlt is bad news as it’s borrowing the OS’s memory and never freeing it. The OS expects for applications to borrow memory and resources, but they should deallocate/Free it as well.

2021.08.11-02:08:20 UTC (local time of the PC where this was collected)

Memory: 3669556K Avail: 1492408K PageFlts:4060786 InRam Krnl: 9468K P:77912K

Commit:2783988K Limit:4586800K Peak:3077384K Pool N:536064K P:84252K

Tag Type Allocs Frees Diff Bytes Per Alloc

LEAK Nonp 340 0 340 356515840 1048576

MxGn Nonp 7985 6436 1549 55044672 35535

2021.08.11-01:57:13 UTC (local time of the PC where this was collected)

Memory: 3669556K Avail: 1529880K PageFlts:2763623 InRam Krnl: 9480K P:67448K

Commit:2544788K Limit:4586800K Peak:2546588K Pool N:372744K P:74848K

Tag Type Allocs Frees Diff Bytes Per Alloc

LEAK Nonp 200 0 200 209715200 1048576

MxGn Nonp 4750 3201 1549 55044672 35535

Clue stepped up and in the TagstoDrivers Folder even tried to help identify what files might be using the tags.

So, I stopped the service that was using LeakyFlt.sys and BAM the problem is solved.

Would you like a way to automate your data captures? Would you like a way to gather data on intermittent issues? Ongoing Issues? Reproducible Issues? Tired of manually setting up tools for data capture? Or can’t seem to stop a trace in time? Then CLUE may just be the tool for you.

Difficulty Generating a Memory Dump

TeedaN — Fri, 22 Aug 2025 01:04:42 GMT

Hi there!

My name is Teeda, and I am a Support Escalation Engineer on the Windows Performance Team at Microsoft. This blog post provides several suggestions and workarounds when there is difficulty generating a memory dump for bugcheck issues (or even hang scenarios). Special thanks to my colleague, Alisse, for assembling this documentation.

Think about the goal…

Is a bugcheck occurring and you are trying to get a memory dump from that? If so, you can skip the parts about manually triggering a dump. However, you may want to use these settings to test out if you can get a memory dump. This will be faster than waiting for the next bugcheck.

Do you need to crash the machine manually? If so, pay attention to the type of machine (virtualized or physical) and the situation we are working with.

Is this a virtual machine?

VMware machines allow to create a snapshot which can then be converted to a memory dump. Often, this is easier than trying to generate the memory dump manually.

Capture the snapshot in the VMWare console with “Take Snapshot” either at the bugcheck screen or if another issue, at the time of the issue.
You will need to contact VmWare vendor to request copy of the vmss2core-sb-8456865 snapshot to memory dump conversion tool as it is no longer publicly available.
Once you have the file, save it on the C drive to a folder called c:\Snapshot
Copy the vmss or vmsn/vmem file that you wish to convert to that folder.
Open an elevated command prompt and run the following command:
1. cd c:\Snapshot
  - For VMs OS until Windows 7/2008R2 use: vmss2core-sb-8456865 –W <snapshot.vmsn/Suspend.vmss> <snapshot.vmem>
  - For VMs OS Windows 8.1/2012 and above use: vmss2core-sb-8456865 –W8 <snapshot.vmsn/Suspend.vmss> <snapshot.vmem>
2. Replace the '<snapshot.vmsn/Suspend.vmss> <snapshot.vmem>’ with the name of the snapshot.
3. This process may take a few minutes depending on the size of the snapshot, but it will create a memory.dmp file in the c:\snapshots folder.

There is also the option to use the NMI switch in VMWare as an alternative if taking a snapshot is not an option. Please note you will still need to configure for a memory dump whether it be kernel or complete: https://kb.vmware.com/s/article/2149185

Hyper-V Machines allows injection of NMI to bugcheck the VM, or you can take a checkpoint of the machine which can then be opened up in a debugger.

To do this for injecting NMI, first ensure the system is configured for the type of dump you need, pagefile is large enough to accommodate the dump size, and enough free space on the drive for where the memory.dmp file will be written.
Next open PowerShell and execute the following command: debug-vm "VM to Debug" -InjectNonMaskableInterrupt -Force
Note: Replace “VM to Debug” with actual guest VM name without the quotes

To do this for VM Checkpoint, in Hyper-V manager select the host machine left pane, right-pane right click the VM you want to create a checkpoint for. Choose "Checkpoint" (or "Snapshot" in older versions). Alternatively, you can click "Action" in the menu bar, then "Checkpoint".

The config and .vmrs files are not stored in the same folder with the virtual disks and differencing disks

The virtual machine folder should contain Snapshots folder and is where the checkpoint (formerly known in Hyper-V Manager as a snapshot) is kept. Collect the .vmrs file

If you cannot locate the .vmrs file, from PowerShell you could run couple of commands to locate.

Get-vmhost should show paths or Get-VM <vmname> | fl VMName,ConfigurationLocation

The VMRS files are stored in the ConfigurationLocation

You will need to engage Microsoft to open .vmrs in debugger for analysis

Alternatively, you can also configure for a manual Hyper-V bugcheck by using:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\hyperkbd\crashdump
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\hyperkbd\Parameters

Configuration information found here: Forcing a System Crash from the Keyboard - Windows drivers | Microsoft Docs

For Azure machines, Azure engineers can grab a memory dump or use NMI:

Configure for complete memory dump:

Step 1: Change page file size

Verify the machine has enough free space for 2x the RAM before continuing.
Launch File Explorer, then right-click This PC. Select Properties
Click Advanced system settings on the System page. Make sure you are on the Advanced tab.
Click Settings under the Performance area.
Click the Advanced tab, and then click Change under the Virtual memory area.
- Note: To enable the system partition, you must uncheck “Automatically manage paging file size for all drives check box.”

Select the C:\ drive for page file location.
Click Custom Size. Set the value of Initial size and Maximum size to the amount of physical RAM that is installed plus 256 megabyte (MB) under the Custom Size button. (RAM*1024 + 256MB = Size in MB)
Click Set, and then click OK.

Step 2: Configure for a complete memory dump file

Go back to Advanced system settings page
Click Settings under the Startup and Recovery, and then make sure complete memory dump is selected.
- Note: If you want to enable the complete memory dump option, manually set the CrashDumpEnabled registry entry to 0x1 under the following registry subkey and restart Windows: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\
Ensure the path is C:\Windows\MEMORY.DMP (%SystemRoot%\MEMORY.DMP)

Click OK
Reboot VM for settings to take effect

Step 3: Enable Boot Diagnostics for NMI Crash

Note: Serial Console requires boot diagnostics enabled

So, if not enabled, go to Boot Diagnostics > click Settings > Turn On > Save

Step 4: Send NMI during issue

When computer is in problem state > Serial Console > click Send Command [1] > click Send Non-Maskable Interrupt (NMI) [2]

Click Send NMI

Dump will be generated.

After completes login to VM and dump will be in C:\Windows\MEMORY.DMP

For AWS machines, try using these steps: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/diagnostic-interrupt.html

For Nutanix machines, please engage the vendor to capture the memory dump.

Do you have the correct configuration?

Step 1: Change your page file size

Verify the machine has enough free space for 2x the RAM before continuing.
Go to Advanced system settings
On the System page, click the Advanced tab.
Click Settings under the Performance area.
Click the Advanced tab, and then click Change under the Virtual memory area.
- Note: To enable the system partition, you must click to clear the Automatically manage paging file size for all drives check box.
Select the C:\ drive for pagefile location.
Click Custom Size. Set the value of Initial size and Maximum size to the amount of physical RAM that is installed plus 256 megabytes (MB) under the Custom Size button.
Click Set, and then click OK three times.

Step 2: Configure for a complete memory dump file

Go back to Advanced system settings
On the System page, click the Advanced tab.
Click Settings under the Writing debugging information area (Startup and Recovery), and then make sure complete memory dump is selected.
- If the complete memory dump is not an option here, to enable the complete memory dump option, manually set the CrashDumpEnabled registry entry to 0x1 under the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl

Step 3: Apply the settings

Ensure there is more space available on the C drive than there is RAM on the machine.
Please restart the machine for the settings to take effect

More Options

Try to use DedicatedDumpFile.sys - How to use the DedicatedDumpFile registry value to overcome space limitations on the system drive when capturing a system memory dump | Microsoft Docs

Manual Dump Trigger Options

NMI

Does this machine have a NMI switch? This would be in the Integrated Lights Out (iLO) Web interface. Create a DWORD value called NMICrashDump under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl and set it to a 1. Then reboot the machine for the setting to take effect.

Keyboard initiated

For a USB keyboard, create the following registry entry:

In HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\kbdhid\Parameters, create a value named CrashOnCtrlScroll, and set it equal to a REG_DWORD value of 0x01.

For a PS/2 Keyboard, create the following registry entry:

In HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\i8042prt\Parameters, create a value named CrashOnCtrlScroll, and set it equal to a REG_DWORD value of 0x01.

Then reboot the machine for the setting to take effect.

Note: you will need to use the Right Ctrl key + press the ScrLk key twice to trigger the dump with the above settings. If the machine does not have those available, there are other options. Forcing a System Crash from the Keyboard - Windows drivers | Microsoft Docs

Ex: Left Ctrl + Space Bar:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\kbhid\CrashDump

Create DWORD value Dump1keys set to 20 (hex)

Create DWORD value Dump2key (note no s here) set to 3d (hex)

NotMyFault

Use NotMyFault to initiate a crash: NotMyFault - Windows Sysinternals | Microsoft Docs

Change the Settings

Ensure there is enough space to capture the memory dump. We need enough space for the page file, and for the memory dump itself which will be the size of the page file.
Disable the Autoreboot:(HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\AutoReboot)
Change the memory dump location to another spot on a local drive
Ensure the option "Overwrite Any Existing File" (found in Control Panel System) is selected. It is a good idea to leave this box checked and to move or copy the current Memory.dmp file.

There is dump logging

You can create a DWORD registry key HKLM\SYSTEM\CurrentControlSet\Control\CrashControl\EnableLogFile set to 1. You will need to crash the machine twice, then you will see a dumpstack.log file on the root of the C drive which will keep track of what occurs during the action of writing to the page file.

Is ASR enabled?

Hardware vendors, such as HP, IBM, and Dell, may provide an Automatic System Recovery (ASR) feature. You should disable this feature during troubleshooting. For example, if HP and Compaq's ASR feature is enabled in the BIOS, disable this feature while you are troubleshooting to generate a complete memory.dmp file. For the exact steps, contact your hardware vendor.

Antivirus and Encryption

Check for any dump filter drivers.
Remove the encryption to test.

What else?

It is possible the paging file on the boot drive is not large enough. To use the "Write Debugging Information To" feature to obtain a complete memory dump file, the paging file on the boot drive must be at least as large as physical memory + 100 MB. When you create a kernel memory dump file, the file is usually around one-third the size of the physical memory on the system. Of course, this quantity will vary, depending on your circumstances.
Also possible there is not room for the Memory.dmp file in the path specified for writing the memory dump.
It is possible that the SCSI controller is bad, or the system crash is caused by a bad SCSI controller board.
If you specify a non-existent path, a dump file will not be written. For example, if you specify the path as C:\Dumpfiles\Memory.dmp and no C:\Dumpfiles folder exists, a dump file will not be written.
Is the Host Guardian Service enabled on either the host or the guest? There are several settings which may prevent dumps from writing. Managing the Host Guardian Service | Microsoft Docs

Grab that Page file!

Ensure the Autoreboot key is set to 0, and when the bugcheck occurs, boot into winre. Grab the pagefile.sys and rename to memory.dmp

Change Altitude of Process Monitor (ProcMon)

TeedaN — Mon, 08 Feb 2021 18:52:45 GMT

My name is Susan and a small group of us have joined together to provide you documentation on how to view a kernel filter driver in procmon on the stack, that is normally obfuscated. A special thanks to my colleague, Becky Burns for documentation collaboration; and a special shout out to Denis Pasos and Ron Stock for both creating a leaky kernel filter driver, and documentation collaboration.

Symptoms or Error

If you need to get Procmon's filter to run below us in the filter stack, it has a setting for that. Procmon is typically used to figure out what is happening on the machine, but you do not get to see the activity of things such as virus scanners because they happen at a lower level than the procmon filter. In our case, we have a driver called Leakyflt.sys but in procmon it only shows as FLTMGR.sys but we want to know which driver it is without performing more tracing.

From an administrative command prompt, we see the driver LeakyFlt at altitude 372000:

In this example below, you will see Procmon’s altitude at 385200 as well as Legacy Filter Drivers such as vdorctl, and dgmaster:

From Procmon, in the stack it looks like

Solution

Changing the "Altitude" that procmon will run, putting it lower in the filter stack. In doing so, you will be able to see all the activity that you want from most filter drivers.

To change the altitude of procmon, you will want to perform the following steps:

Install Procmon (assuming you have not already installed it) https://docs.microsoft.com/en-us/sysinternals/downloads/procmon
From an Administrative Command prompt, run FLTMC to see the Altitude of the filter drivers:
In the screenshot, the lowest filter driver altitude is 37200
Open Registry Editor (RegEdit)

Navigate to registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PROCMONXX\

In Example: PROCMON24 (name may have a different number on your machine)

Expand to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PROCMONXX\Instances\Process Monitor XX instance. (i.e., Process Monitor 24 Instance)
Change the Altitude Regkey value to lower than your lowest filter driver.
For this example: change the Altitude value to 40000 (which will show you virtually everything that is happening on the machine). Alternatively, you could set the altitude to 372000 if you suspected a specific driver.

Ex. Default altitude currently set to 385200

Right click on Altitude, change value to 40000, click OK
You must also set the security on the "Process Monitor XX Instance" key and add deny rights for everyone for "delete" and "set value". Reason being that procmon will try to change its value back right away. You will have to select "Disable inheritance" to be able to set them at the Process Monitor XX Instance level.

Right click on Process Monitor 24 Instance, select Permissions…

Click Add

In “Enter the object names to select:” type Everyone, click Check Names, then click OK.

Select Everyone, Click Advanced.

Select Everyone, click Edit.

Click Show advanced permissions.

Change Type: to Deny, check Set Value, check Delete, click OK.

(if Read Control is checked, uncheck it)

With Everyone highlighted, select Disable inheritance, click OK

Choose Convert inherited permissions into explicit permissions on this object.

Click OK.

Click Yes

Click OK

Exit Registry Editor
If you have already started procmon before doing these changes, you will need to restart the machine. If not, you should be able to just start procmon.
From an elevated command prompt, run the command fltmc instances and verify that the procmon drivers are running at the altitude that you set (ex. 40000).

Then reproduce the scenario you want to capture. Your capture will be even larger than normal.
Notice now when we review the new procmon, and view Stack we see the name of the driver LeakyFlt.sys.
Note: You can leave the setting as it just lowers the threshold of what we see. And more is always better when it comes to legacy kernel drivers. Once you get a procmon with that enabled, you can look at the stack and see it.

Note 2: Will the deny permission for Everyone only impacts that instance? Will it not interfere with other applications/permissions on the machine?

It only affects that procmon instance; not all procmons. So, if they installed e.g., something that had their own procmon instance it would not impact it. You can take ownership of the key to delete the key when you are done.

Windows 10/Server 2016 Graphics Troubleshooting

TeedaN — Wed, 03 Feb 2021 17:22:13 GMT

My name is Susan, and I am a Support Escalation Engineer on the Windows Performance Team at Microsoft. A special thanks to my colleague, Dan, for his input.

This blog will address how to troubleshoot graphics issues in Windows 10 and includes identifying and data collection.

Scenario

You have a Windows 10, Server 2016 or later showing graphics corruption such as black bars*, trails*, artifacts* when moving windows, or distortion. (*Defined at end of this post.)

Troubleshooting

When you experience graphics corruption, always start by isolating it to a specification application, OS, and identify steps to reproduce the issue.

First, confirm if the issue is specific to an application. If it’s only a custom in house application, or a 3^rd party application, please engage the vendor of the application.
Take a screenshot of the distortion, or a video demo’ing the distortion (use your cell phone if necessary).
In this example, let’s assume Word is the only application impacted:

Test starting the application in Safe mode (if applicable)
- Start > Run > Winword /safe

- Hold down the CTRL key while starting the application until the following prompt appears:
If the issue does NOT reproduce, begin investigating Add-ins that may be impacting the application:
Start the application normally, and Enable Hardware Acceleration
- Microsoft Office 2019/16/13 is a great software for Windows 10/8 if application specific (e.g. Office) test Enabling the Disable Graphics Acceleration
- Click File
- Click Options
- Click Advanced
- Scroll down to the Display Section and Enable Disable Hardware graphics acceleration (screenshot shows it is currently disabled)

If the issue is more systemic:

Update the Graphics Card driver/GPU to the latest drivers and reboot
1. Use Device Manager and click on Display Adapters
2. Select the adapter, right-click and select Update driver
Test using a lower video resolution on the OS or the Base video
1. Start > Search > MSConfig
Boot into Windows Safe Mode with network and test
1. Start > Search > MSConfig
In many versions of Windows 10, WPR is built in. Use either following the GUI or command line below
```
wpr -start gpu -start video -start audio
```
1. - Reproduce the issue e.g. run a workload such as video playback or a real time communications scenario, or dragging an application across the screen.
  - Run the following command:
```
wpr -stop Media.etl
```
Run DxDiag
1. Start > Search > DxDiag
2. Check the Video drivers and confirm they are up to date. If an update was just applied, you may wish to rollback and test.
3. Lastly, Save All
Last resort: gather a complete memory dump of the Operating System while the distortion is present.

*Glossary and Guide: graphics issues

Black Bars: desktop may appear to have black lines either vertical or horizontal.
Trails and Artifacts: On moving a tiled window or application, it may leave a trail of what appears to be orphaned.

Alternative Tools for Application Hangs

TeedaN — Fri, 18 Sep 2020 18:40:33 GMT

My name is Susan Buchanan and today I’d like to discuss some basic troubleshooting for applications hangs using Task Manager, Resource Monitor, and WaitingOn outside of getting an application dump. A special shout out to Leo Fagundes for writing the WaitingOn application.

Troubleshooting application hangs is key to resolution. Getting an application dump will not always be a smoking gun and should be used as a last resort for various reasons as missing symbols for 3^rd party applications or waiting on other processes.

Task Manager & Resource Monitor – Analyze Wait Chain: Beginning with Windows 8, in 2014 we added new options into Task Manager as well as Resource Monitor was the “Analyze Wait Chain” option when you right-click on a task in the Details view. This allows you to see what processes are waiting for a resource that is being used by another process.

What this means is that if you have an application hanging for some reason, you can analyze the wait chain to see whether it is waiting on something that is in use.
Additional Information on how to prevent hangs in windows applications:
https://docs.microsoft.com/en-us/windows/win32/win7appqual/preventing-hangs-in-windows-applications and a Blog which talks about the “analyze wait chain” feature:
https://blogs.technet.microsoft.com/tip_of_the_day/2014/03/01/tip-of-the-day-wait-chain-analyzer/

WaitingOn.exe a simple tool that helps troubleshoot hung applications and services.

WaitingOn.exe is available at https://github.com/leonardomsft/WaitingOn/releases/download/v1.1/WaitingOn.exe, and besides functional, it’s still experimental. Use it with caution. If you install it on a customer’s machine, please remove it at the end of the troubleshooting session.

WaitingOn.exe displays all the blocked threads from a process and what they are Waiting On. It can also display all blocked threads for all processes in the system.

WaitingOn.exe leverages the Wait Chain Traversal (WCT) API introduced in Windows 2008/Vista. The WCT API is the one behind the "Analyze Wait Chain" functionality in Task Manager and Resource Monitor.

Task Manager

One of the new options added into Task Manager in recent versions was the “Analyze Wait Chain” option when you right-click on a task in the Details view. This allows you to see what processes are waiting for a resource that is being used by another process.

What this means is that if you have an application hanging for some reason, you can analyze the wait chain to see whether it is waiting on something that is in use.

For instance, we printed from Word, and then used this option while the print process was happening to see what would happen. In this case, Word was waiting for splwow64.exe, which handles printing from 32-bit applications.

It’s worth noting that because Word is written properly, the GUI interface doesn’t actually hang while it is waiting for the other process.

In this case the process is working correctly and is not hung.

For 3^rd party applications, it may show you other processes it is waiting on. If the selected process is waiting for another process, a tree organized by dependency on other processes will be displayed. (See screenshot above).
NOTE: Many system processes depend on other processes and services for normal operation. Both Task Manager and Resource Monitor will display wait chain information for any process.

In a second scenario BadApp.exe is not responding

We attempt to Analyze Wait Chain from within Task Manager

If the application is not waiting on anything, it will be empty:

If the application is waiting on something, it will show the process it is waiting for:

(In this example you can see that Perfmon is waiting on a svchost). We will dig deeper into this in a few moments.

Resource Monitor

Use case example: Badapp

Start Resource Monitor by either typing ResMon from the search box or starting the Resource Monitor icon from the Start Menu.

In Resource Monitor we see badapp.exe is not responding.

If a process entry in the table is not red, if the process status is Running, and if the program is operating normally, then no action should be required by you.
If a process entry in the table is red, if the process status is not running, and if the program is not operating normally, then you can try killing the process it is waiting on starting with the child processes in red first, then moving to the parent processes.

You can start a “wait chain analysis” from Resource Monitor as well, simply do a right click on the process you want to investigate.

Using WaitingOn.exe & it’s Advantages

WaitingOn.exe has the following advantages over Task Manager and Resource Monitor:

It displays what type of object is blocking the thread.
It displays the name of the object blocking the thread.
It can be scripted.
It can be run against a computer that you can't logon (by using PsExec or Remote Powershell).

WaitingOn.exe was built on top of the sample WCF code available in the MSDN documentation.

Use case example: Badapp

As seen earlier, this process is the parent process that is hung and is not waiting on any other processes thread.

Use case example : Perfmon

In this example, we had a hung perfmon.exe window that was not responding to mouse clicks. Attempting to run the Analyze Wait Chain command from Task Manager produced the following results:

We can see that perfmon.exe (PID 3292) is waiting on an svchost process (PID 1564), but since it hosts many services, there is not much we can do.

<background information>
If you encounter a service host with multiple services, always consider isolating the services. In Windows 10 / WS 2016 we have introduced the change by default (if you have more than 4 GB RAM). To isolate a service, we use the sc.exe command line tool:
sc config <servicename> type= own
Important: there is a space between type= and own which is required!
https://docs.microsoft.com/de-de/windows-server/administration/windows-commands/sc-config
</background information>

Now, running WaitingOn.exe against the hung perfmon.exe revealed that Thread 7352 was blocked by an Alpc called “\RPC Control\DNSResolver”, which was found in the svchost.exe handle list:

We then can use Process Explorer to further investigate or kill that object (which might result in unexpected behavior!)

In this case we decided to simply restart the associated service (DNS Cache service) which unfroze the perfmon.exe window.

Additional information:

Links within this article:

Additional Information on how to prevent hangs in windows applications:
https://docs.microsoft.com/en-us/windows/win32/win7appqual/preventing-hangs-in-windows-applications
and a Blog which talks about the “analyze wait chain” feature:
https://blogs.technet.microsoft.com/tip_of_the_day/2014/03/01/tip-of-the-day-wait-chain-analyzer/
WaitingOn.exe at github: https://github.com/leonardomsft/WaitingOn/releases/download/v1.1/WaitingOn.exe,
Wait Chain Traversal:
https://docs.microsoft.com/de-de/windows/win32/debug/wait-chain-traversal
sc command line options:
https://docs.microsoft.com/de-de/windows-server/administration/windows-commands/sc-config

MYSTERY MEMORY LEAK: WHERE DID MY MEMORY GO?!

TeedaN — Wed, 16 Sep 2020 21:06:47 GMT

Hello,

My name is Jeffrey Worline, and I am a Senior Support Escalation Engineer on the Windows Performance Team at Microsoft. This blog addresses how to troubleshoot unaccounted memory usage or leak to include identifying and data collection.

If you already determined the process consuming memory, check out my previous blog post: Memory Leaks in a Process

Scenario

When you cannot reconcile the amount of RAM being used with task manager, resource monitor, or perfmon collection.
Large chunk of RAM being used but you cannot see where or by what.

Troubleshooting

Scenario A

When large amount of RAM is being used by not accounted for in task manager or resource manager. How do we find or account where that mystery memory is being used? RAMMap from Sysinternals is the tool needed for the job.

First, when looking in task manager and at the memory usage by processes to view memory usage, ensure you also look in the Memory box on the performance tab – the amount of cached, paged pool, and non-paged pool memory usage.

Download RAMMap
Launch RAMMap to have it take a snapshot of memory usage.

Glossary and Guide to the column and row headings

Stages of memory

Active: Pages of physical RAM in active use by the specified category (usually a process working set or the system working set).
Standby: Pages of physical RAM not actively being used. These are still left in physical RAM but will be repurposed first by the memory manager (either returned to the active list or zeroed out and reused) if something needs physical ram for active pages.
Modified: Similar to Standby, but these are pages of physical RAM that have been changed and must be flushed to disk before reusing them.
Modified no write: Similar to modified pages but have been marked not to write out to disk.
Transition: Pages that are in transition between any of the other categories
Zeroed: Pages that have been zeroed out and are ready to be used – they can be quickly allocated for new physical memory allocations
Free: Free pages are free to be used but have some type of “dirty” data in them so they must be zeroed for security reasons before given to a user process. These are usually pages that have been freed by an existing process.
Bad: These are physical pages that have been marked as bad.

Areas of interest would be the following rows to check for high memory consumption to account where the rest of your memory is being used.

TIP:

If you have a memory leak and get to the point of almost running out of memory, the normal procedure is to reboot the machine in order to clear out the memory. You can use RAMMap to clear areas of memory negating the need to reboot the machine.

Types of memory usage

Process Private: Memory allocated for use only by a single process.
Mapped file: Mapped “views” of files are when the contents of that file are mapped to virtual addresses in memory.
Shareable: Pages that have been marked as shared can be used by multiple processes.
Paged Pool: Kernel pooled memory that can be paged to disk.
Nonpaged Pool: Kernel pooled memory that cannot be paged to disk.
Session Private: Memory that is private to a particular logged in session. This will be higher on RDS Session Host server.
Metafile: Metafile is a part of the system cache containing NTFS metadata and used to increase the performance of the file system.
AWE: You will typically see this used by SQL or other database applications.
Driver Locked: These are pages that have been locked in physical RAM by a driver. Usually see this usage with Hyper-V or VMware virtual machines.
Large Page: Normal page size for Windows memory is 4kb on x64 systems. But with large pages, the size is 2mb. SQL Server and Oracle support the concept of Large Pages when allocating memory.

In this snapshot, you can see that about half of the physical RAM being used is by Mapped Files:

In this example, next we would click on the Physical Pages tab
Now at the bottom of the tool select "Use" for the Filter and "is" select "Mapped File" from the drop down.

This will now show you all the mapped file entries.

Next, I would click on the File Name column heading to group similar file names so at this point I could look to see if all the mapping were going to the same path or general path to help determine what is causing all the mapped files.

This information is not something you will see any place else other than an RAMMap or memory dump.

Scenario B
On a VMWare or Hyper-V system, the hypervisor can take memory away from one VM and give it to another VM. It does this by using a driver loaded in the VM to "lock" the memory at the kernel level which can then be given to another VM. If too much memory is taken away, this will cause working set trimming and general performance issues. Standard perfmon memory counters will not provide the info to account for the missing memory. This driver locked or "ballooned" memory can be seen 4 different ways depending on the OS.

VMWare console - Memory and processor utilization for each VM will be clearly seen in the VMWare console. If you have access to the console, then this is the preferred method to see the state of memory in the VM.

VMware performance counters - When VMWare tools are installed, VMware performance counters are also created. These can be manually loaded in Performance Monitor or use the logman.exe method below to set up perfmon collection.

Example of Logman to collect VMWare processor and memory counter:

The following will configure the counters, set logging to circular with max file size of 300 mb, and take a counter reading every 3 seconds.

The resultant log will be place in c:\perflogs.

<<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

Copy and paste the following command into the command prompt window:

Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short.blg" -f bincirc -v mmddhhmm -max 300 -c "\LogicalDisk(*)\*" "\Memory\*" "\Cache\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Processor Information(*)\*" "\Process(*)\*" "\Thread(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues(*)\*" "\Terminal Services\*" "\VM Processor\*" "\VM Memory\*" -si 00:00:03

Start the log with:

Logman.exe start PerfLog-Short

To stop perfmon log:

Logman.exe stop PerfLog-Short

Example output from Perfmon:

Example Sysinternals RAMMap:

- Jeffrey Worline

Memory Leaks in a Process

TeedaN — Wed, 16 Sep 2020 21:08:53 GMT

Hello,

My name is Jeffrey Worline, and I am a Senior Support Escalation Engineer on the Windows Performance Team at Microsoft. This blog addresses troubleshooting leaks occurring in a process to include identifying and data collection. This article assumes that you have already identified the process that is leaking. If you have not yet been able to identify where your leak is, please see my blog: MYSTERY MEMORY LEAK: WHERE DID MY MEMORY GO?!

First thing we need to determine is memory consumption being caused by private data, heap data or some other memory type. We need to address the memory types differently.

Download a Windows Sysinternals tool called VMMap

VMMap is an utility application analyzing virtual and physical memory. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types.

This tool is used to attach to an individual process allowing a snapshot to be taken to see the memory map for that process.

Simply launch VMMap and from the process list it displays, pick the instance showing the high private working set.

If the high memory is being caused by Heap, you will need to enable User Stack Tracking (UST) against the process using gflags.exe which is part of the Debugging Tools for Windows.

Note: If the high memory shows as Private Data or some other type other than heap, simply continue with getting procdump when memory usage is high.

Disclaimer: The intent of this blog is not to teach you how to debug. If you are not familiar with debugging process dumps, please open a case with Microsoft support for assistance

Scenario A: Uniquely named process with high memory by Heap

Download Debugging Tools for Windows

Just need the standalone version since we only need the debugging tool and not the whole WDK package.

Note: If the high memory shows as Private Data or some other type other than heap, simply continue with getting procdump when memory usage is high.

Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right-click on it and select “Run as administrator.”
Click on “Image File” tab.
Type in the process name, for example notepad.exe

Hit the keyboard TAB key to refresh
Place check mark in “Create user mode stack trace database.”

Note: Be sure to reverse your gflag setting also by unchecking the “Create user mode stack trace database” when no longer needed.

Click “OK”.
The process or service will need to be restarted to put in to effect.
Get procdump of process when memory is high.

Scenario B: High memory occurring in svchost.exe process by Heap

If the svchost process contains more than one service, you will need to break each service out to run in its own svchost process to determine which service is causing the high memory. Once that has been determined, then need to uniquely name that svchost process the service runs in and then enable UST against it. You do not want to enable UST globally against all svchost process as it will put a serious performance hit.

Note: We don't ever want to enable UST against svchost.exe as that would enable against any and all instances of svchost.exe running and could cause a performance hit on the machine because of the overhead.

Use Task Manager to document the PID of the service that is demonstrating high memory usage.
From administrative command prompt run following command:
```
tasklist /svc
```
Using PID you documented from Task Manager, locate that svchost process and document the services that it is hosting.
Break each service out into its own svchost process if it is a shared svchost process hosting several services by running following command for each service:
```
sc config <service name> type= own 
```
replace <service name> with actual service name
Note: there is no space in “type=” and there is a space between “= own”

Restart the service for setting to take effect.
Verify the services are running in their own svchost process by running tasklist /svc from command prompt again.
At this point, you have broken each service out into its own svchost process; now identify which service was driving up memory usage before proceeding to next step.
Once the service has been identified, from administrative command prompt change command focus to c:\windows\system32 folder if needed and run following command:
```
copy svchost.exe <unique name>
```

Replace <unique name> with something that represents the service. Example for wmi service - wmisvchost.exe

Launch registry editor (Start > Run > “regedit.exe”) and navigate to HKLM\System\CurrentControlSet\Services then the appropriate key for the service you are uniquely naming svchost process for.
Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\<unique name> -k netsvcs

Replace <unique name> with that used in step 8. In this example that would be: %systemroot%\system32\wmisvchost.exe -k netsvcs

Note: Backup the registry key before modifying it

Restart the service.
Use gflags as noted earlier to enable “Create user mode stack trace database” against the uniquely name svchost process, then restart the service to apply the new settings.

Note: It is important that you go back and reverse what you did in step 4 and modify path back to original after you are no longer needing the service to be broken out and uniquely named as failure to do so can prevent future hotfixes from being installed associated with that service.

To reverse, replace sc config <service name> type= own with sc config <service name> type= share

Reverse your gflag setting also by unchecking the “Create user mode stack trace database”.

Reverse your setting in the registry under the service key for the ImagePath.

Get a procdump of the process when memory is high.

Directions for Procdump

Download Procdump tool from Sysinternals

Note: Use procdump if dumping 32-bit process and use procdump64 if dumping a 64-bit process

From and administrative command prompt, navigate to the directory where you downloaded and unzipped procdump tool.
Run the following command to dump a unique name process:
```
procdump -ma <process name>
```

The -ma switch is to perform a full memory dump of the process.

Replace <process name> with actual process name.

If there is more than one instance of a process running with the same name you must dump the process by PID as opposed to name.

e.g. procdump -ma <PID>

Replace <PID> with actual PID of the process.

Next Up: MYSTERY MEMORY LEAK: WHERE DID MY MEMORY GO?!

- Jeffrey Worline

How to configure file associations for IT Pros

CraigMarcho — Wed, 15 Feb 2023 19:03:02 GMT

All steps described in this have been tested on Windows 10 versions.

These steps apply to Windows 10 release through Windows 11.

Configuring file association prior Windows 10

There were different ways and guidance to set default program prior to Windows 10 (see Managing Default Applications).

Before Windows 10, an application could check default apps, ask for user consent and set default app programmatically using Windows API.

But some programs skip the user consent and set the app defaults into the registry. The main requirement for default file association is often forgotten: the end-user is in control.

Now in Windows 10 checks if registry file extension keys have not been modified to prevent file association hijacking.

File association changes in Windows 10

The way that default file associations work changed in Windows 10.

There is a new UI for the end-user. This new way puts the user in control with a new file association notification.

This notification will be displayed:

On the first launch of a file extension, if multiples programs are registered for handling that file extension.
Each time a new application registers a file extension, except if the Always use this app to open .xxx files is checked.

If an application used Windows API to set default apps, the user will receive the following notification:

For more information about these changes : https://blogs.windows.com/windowsexperience/2015/05/20/announcing-windows-10-insider-preview-build-10122-for-pcs/

You will find some explanation on The Old New Thing blog: Why do my PDF file associations get reset every time I restart?

Windows 10: An app default was reset

But what happens if an application is not using Windows API and writes some stuff and hijacks user preferences?

Now,Windows 10 detects that the registry is corrupted and will reset the default program for this file extension. Additionally, the end-user will receive this notification: An App default was reset. These change is documented in KB4001770: Reset app default when a registry setting is deleted or corrupted and streamlined notification about the corruption.

How to configure file association in Windows 10?

It's possible for IT Pro to configure or force default association using supported methods.

The best way to do it is to set up a reference computer, install applications, configure default programs and use Dism to export/import the custom default app associations or use a group policy.

Set up a reference computer
Install applications
Go to Control Panel\All Control Panel Items\Default Programs and configure default apps associations. In Windows 10 1709, this control panel item is now in Settings app.

Let's try to configure Internet Explorer as the default browser

Choose Internet Explorer

Internet Explorer is now the default web browser

Export/import the custom default app association with dism.exe
Note that you need administrator rights to use dism.exe. And to export properly the associations use the same account used in step 3, otherwise you will get a malformed XML file.

Dism.exe /online /export-defaultappassociations:C:\temp\CustomFileAssoc.xml

PS C:\Windows\system32> dism /online /export-defaultappassociations:"C:\Temp\IE-DefaultBrowser.xml"

Open the xml file and check if everything looks good.

Usually at this step you will be tempted to delete other lines because you simply don't care about them and get a file like this:

FTA-IE-Only.xml

VERY IMPORTANT: If you want to import your file with DISM.exe, DO NOT delete any file associations entries!
A missing entry will trigger the App default reset notification and you will get a notification storm at the first logon.

Refresh your XML on a regular basis

As some recommended applications can manage more extensions with each new Windows 10 version available, it's a good practice to refresh your XML. For example, in Windows 10 1703, Microsoft Edge registers the epub extension. If you're using an XML file from Windows 10 1607, epub is missing. As a result, you will get an app reset notification for epub.

Tips for building your XML file

Manually editing the file could result in a non-valid XML file. Ensure that your XML file is valid. Opening XML file in the old Internet Explorer is a good idea to check if the XML is valid. You can try XML Notepad to edit/validate XML files.
If you do not see your file extension in XML file, go back to Control Panel\All Control Panel Items\Default Programs and configure default apps associations, select file extension, click on Change Program and confirm the program in the dialog box. Then, export again you're XML file.

Deploy your custom XML

Now it's time to apply your XML file. You have two options:

Set up file association in your Windows 10 image. File associations will be configured for new users' profiles. Existing profiles are untouched. Users can change file associations.
Configure a policy for your domain-joined computer: file association will be configured at each logon. User will be able to change file association, but at the next logon file association will be configured using XML file. This policy works only for domain-joined computer.

Configure the XML file for your Windows 10 reference image:

Dism.exe /online /import-defaultappassociations:c:\temp\CustomFileAssoc.xml

Your file will be copied in \Windows\System32 with the following name OEMDefaultAssociations.xml

Configure the XML file for your domain-joined computer

Configure the following policy Set a default associations configuration file located in Computer\Policies\Administrative Templates\Windows Components\File Explorer.

If this group policy is enabled and the client machine is domain-joined, the file will be processed, and default associations will be applied at logon time.

Note: this policy will not prevent user to change the file association. But at each logon the default association configured in XML file will be applied.

How to force only a set of file associations?

Some IT Pros want to force only some file associations and let users in control for others file associations. For example, they want to configure Internet Explorer as default for HTTP/HTTPS only for HR people.

So, they removed everything in the XML file except Internet Explorer entries. As previously seen, removing entries in XML file could result in app reset notifications.

But don't panic, there is a solution to do it in your Windows 10 reference image.

You just need to have two XML files, one for configuring the defaults and another one to force file association.

The main XML, CustomDefaultAssoc.xml must contain all extensions. You need to import this file using:

Dism /online /import-defaultappassociations:C:\CustomDefaultAssoc.xml

The second XML, FTA-IE-Only.xml will contain only a set of file extensions. You need to use this XML file with the group policy Set a default associations configuration file.

At the first logon, Explorer.exe will apply both XML.

The end user will have this results without any app reset notifications:

PDF files associated with Microsoft Reader.
HTM/HTML files associated with Internet Explorer.

OEMDefaultAssociations.xml contains the following lines:

IEOnly.xml contains the following lines:

Why I'm getting an app reset notification?

Raymond Chen talked about this problem. If a program is trying to set some registry keys, Explorer will detect it and the file association will be reset.

Stop using script or other pre-Windows 10 ways for configuring file association.

You can check the Microsoft-Windows-Shell-Core/AppDefaults event log for clues about file associations reset.

If you want to troubleshoot this, keep calm and run Procmon
😉

You should be able to track which application is hijacking your file associations. Once you identified the bad application, the best way to solve this, is to use a more recent version, or contact your vendor.

Why I'm getting the User Choice notification even if I used an XML file?

You could get the following notification starting Windows 10 1703 when you choose an application as the default viewer instead using Microsoft Edge.

These notification windows are displayed only once if you're clicking on OK. It's because Microsoft Edge is detected as a new application.

If a new app is installed and is registered to an existing file extension or protocol you will get this notification.

If you want to get rid of these notifications, there is a group policy to hide these notifications:

Do not show the 'new application installed' notification

This policy removes the end-user notification for new application associations. These associations are based on file types (e.g. *.txt) or protocols (e.g. http:) If this group policy is enabled, no notifications will be shown

You can find in the table below some group policies related to default file associations settings:

Table 1
Policy Setting Name	Policy Path
Do not reinitialize a pre-existing roamed user profile when it is loaded on a machine for the first time	Windows Components\File Explorer
Turn off Internet File Association service	System\Internet Communication Management\Internet Communication settings
Turn off access to the Store	System\Internet Communication Management\Internet Communication settings
Do not show the 'new application installed' notification	Windows Components\File Explorer
Set a default associations configuration file	Windows Components\File Explorer

Bonus for Adobe Reader XI users

If you are still using Adobe Reader XI, you should get a new application to open PDF, as Adobe Reader XI is out of support.

With Adobe XI installer, you can get an app reset notification for PDF extensions and it will be reset to Microsoft Edge.

You should look at Adobe Reader documentation and launch the Adobe Reader 11 installation with the following command line:

AdbeRdr11010_en_US.exe /rs /sAll /msi EULA_ACCEPT=YES OWNERSHIP_STATE=0

With OWNERSHIP_STATE=0 Adobe Reader stops to change file association for PDF with an unsupported way.

Resources:

Recent Resolved issue with Windows 10: Search from start show blank and no results

CraigMarcho — Mon, 10 Feb 2020 20:43:00 GMT

Hello AskPerf! My name is Susan from the Microsoft Performance team. We have some important information we wanted to share about a recent issue that has been resolved.

We were made aware recently of a temporary server-side issue causing Windows search to show a blank box. This issue has been resolved for most users and in some cases, you might need to restart your device. We are working diligently to fully resolve the issue and will provide an update once resolved.

https://docs.microsoft.com/en-us/windows/release-information/windows-message-center#387

Help! My Powershell script isn't working! Can you fix it?

CraigMarcho — Mon, 15 Jul 2019 22:21:35 GMT

Hello AskPerf blog, Travis Gradert here from Microsoft Customer Services and Support. I handle quite a number of Powershell cases for our support team, and have noticed that we get a lot of cases where customers are asking for help with things that we do not support. I wanted to write here to help explain what we do and do not handle in our support team when it comes to Powershell. The easy part of this is that we do not support custom or 3^rd party scripts. There is no support for the creation of a script. CSS representatives may put in a best effort to you as the customer, however engineers may not have the expertise required.

Instead of engaging a Customer Services and Support team, if you are a premier customer and have a Technical Account Manager, they should be contacted to engage Microsoft Consulting Services for the creation or debugging of a custom or 3^rd party script.

[note scripts found in the Powershell Gallery are not supported scripts. Issues with scripts need to be addressed with the author, found on the page where the script was downloaded.]

So what is and is not supported? If a specific cmdlet, or the Powershell engine itself seems to be having an unexpected failiure, Customer Support and Services will be able to assist and work towards resolution of the failure. Below are some examples of supported requests.

A specific Powershell cmdlet with appropriate parameters returns:
1. No result
2. An unexpected result (missing or extraneous data)
3. An error
The engine/console fails to load or crashes
Configuration and Customization of the console
Remote Connectivity and Authentication
Microsoft-Provided DSC Resources fail unexpectedly

Examples of Unsupported requests

Create script to perform “X” action.
Review “X” script to determine a cause of a failure. No script debugging has been performed, issue hasn’t been isolated to a specific commandlet.
Create DSC configuration

I’d like to provide some links that I think are helpful when looking for assistance with Powershell Scripts:

Script Debugging Resources

Using ISE to debug scripts

https://docs.microsoft.com/en-us/powershell/scripting/components/ise/how-to-debug-scripts-in-windows-powershell-ise?view=powershell-6

Remote Runspace Debugging

https://devblogs.microsoft.com/powershell/powershell-runspace-debugging-part-1/

Transcription and Logging:

https://blogs.technet.microsoft.com/ashleymcglone/2017/03/29/practical-powershell-security-enable-auditing-and-logging-with-dsc/
https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.host/start-transcript?view=powershell-6

Measuring Command Performance:

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/measure-command?view=powershell-6

Introducing Windows Terminal!

CraigMarcho — Wed, 08 May 2019 00:04:37 GMT

Hey AskPerf community, Craig Marcho here with some cool news....

Kayla Cinnamon, Program Manager for Windows Console, Command Line and WSL, recently announced Windows Terminal, which will be available this Summer from the Microsoft Store.

What's that? You can't wait until Summer? No problem! You can clone, build, run, and test the code from the repository on GitHub:

https://github.com/Microsoft/Terminal

You can read more about this exciting announcement here:

Introducing Windows Terminal

Thanks for visiting, and we'll see you again soon.

Install PowerShell on macOS and Linux

CraigMarcho — Thu, 18 Apr 2019 15:53:03 GMT

Did you know you can run PowerShell on macOS and Linux based operating systems?

Just wanted to pass this on in case you did not know...

For the macOS…

https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-macos?view=powershell-6

for Linux…

https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-6

Now you know!

Is your Windows 2012 R2 server crashing after March 12, 2019 updates with bugcheck 0xd1 or 0xfc?

CraigMarcho — Tue, 16 Apr 2019 00:28:29 GMT

Hello AskPerf! My name is Susan from the Microsoft Performance team. We have some important information we wanted to share right away.

After installing the March 12, 2019 updates (KB4489881 and KB4489883) on Windows 2012 R2 machines, devices with a winsock kernel client may experience 0xc2, D1, FC and possibly other bugchecks. (including event 41)

Systems that run the Skype for Business or Lync Server Edge Transport role may be impacted by this issue.

At this time, we recommend you apply one of the following patches and reboot.

1. Windows 8.1 and Windows Server 2012 R2 Security Only: 4493467 << includes security fixes only

2. Windows 8.1, Windows Server 2012 R2, and Windows RT 8.1 Monthly Rollup: 4493446 << includes additional fixes both non security fixes, and older security fixes.

If you are experiencing this issue, and applying either 4493467 or 4493446 does not resolve the issue, please call Microsoft and open a support incident.

Powershell Core SSH Remoting

CraigMarcho — Wed, 03 Apr 2019 21:47:40 GMT

Hi Everyone, my name is Travis, and I am with the User Experience team here at Microsoft.

Today I’ll be discussing implementing SSH remoting as a Transport for Powershell.

While WinRM is a great transport for Active Directory domains, or domains with a two way forest trust – where WinRM is lacking is in scenarios where there either is no trust between domains, or no domain membership at all. Where there is no domain membership, certificate authentication is your only real choice when it comes to secure Remoting via WinRM.

SSH as a transport changes that because public/private key pairs are used to secure the connection, allowing for encrypted transport of the data sent between the machines – even when an active directory solution has not been implemented. This also means that if you prefer to admin from a *nix installation, you now have a viable transport to manage windows machines from a native *nix remoting implementation.

Newer OS’s include the OpenSSH client natively as a windows capability inbox. The client is enabled by default, however the server is not. To validate the OpenSSH install state, run the following.

Get-WindowsCapability -Online -Name *SSH*

Add the features by piplining get-windowscapability to add-windowscapability. No restart is needed to complete the installation.

On server 2019, be sure to run the latest Cumulative Update, as the error shown below will be thrown due to a known issue that is already resolved in December release patches.

Get-WindowsCapability -Online -Name *SSH* | Add-WindowsCapability -Online

As far as initial configuration is concerned, the service (sshd) must be started, and should be set to automatic as it is not by default. In addition, the firewall rules for SSH must be enabled. This is a TCP connection on port 22.

Start the Service with the below command.

Start-Service sshd

Set the service to automatic startup.

Set-Service -Name sshd -StartupType 'Automatic'

Confirm if the firewall rule is enabled.

Get-NetFirewallRule -Name *ssh*

Set the firewall rule to allow incoming connections if not allowed

Get-NetFirewallRule -Name *ssh* | Set-NetFirewallRule -Enabled True -Direction Inbound -Action Allow

Testing SSH:

Simply use SSH.exe and specify a remote machine name to use implicit credentials against the remote machine to authenticate.

Once the session has been formed, you should simply see a prompt similar to the following:

Simply run the hostname.exe executable to confirm you’re working with the remote machine.

Remoting with Powershell Core

Download and install the latest version of Powershell Core, available on Github. https://github.com/powershell/powershell

For the purposes of this Demo, I’m installing the x64 MSI package on both my Windows 10 1809 client, as well as Server 2019 to the default location.

Powershell Core must be configured as a subsystem in $env:ProgramData\ssh\sshd_config. As a note, there is currently a bug in OpenSSH where paths with spaces fail to be recognized as a subsystem. As a work around you can use a symlink to correct the error and point the sshd_config to the symlink location.

For the stable release, use: mklink /D c:\pwsh "C:\Program Files\PowerShell\6"

For the preview release, use: mklink /D c:\pwsh "C:\Program Files\PowerShell\6-preview"

Note that we’re pointing to the symlinked path in the subsystem config below.

After the service configuration is changed, restart the service.

Restart-Service sshd

Powershell Core adds a few new parameters to the Enter-PSsession commandlet, -Hostname, -SSHTransport, -Keyfile, and -Subsystem. Note that you cannot use -computername in combination with -SSHTransport.

An example of Remoting is shown below.

And that's it for today! Thanks for reading, and let us know if you have any comments or questions below!

Welcome Back AskPerf!

CraigMarcho — Mon, 25 Mar 2019 17:22:00 GMT

Hello Everyone! Welcome to the new home of the AskPerf blog. We have recently moved, and many things have changed in the years since the last post here. The owners of the blog have moved on to bigger and better things, so I have decided to bring it back to life, and bring some new fresh content to help the community. The Performance team has actually been split to two teams internally at Microsoft, one is still called "Performance" and the other is called "User Experience", or UEX. Between the two teams, we will still provide information on all of the topics that we cover, here under one roof. So welcome back, and we will be providing some information here soon.

-Craig Marcho

WOW...are folks still reading this blog???

CraigMarcho — Sat, 16 Mar 2019 12:51:15 GMT

First published on TECHNET on Mar 05, 2018
We haven't posted any blogs for ~2 years, yet it seems that people are still reading/commenting on previous posts. For me, I've moved on to the Security Team here at Microsoft, and there really is no more Performance team. They've been broken up into smaller support teams that support certain topics. From a Security standpoint, I might be able to provide a few posts from time to time regarding the topic if your interested. Thoughts? -Blake

AskPerf Blog transition…

CraigMarcho — Sat, 16 Mar 2019 12:51:11 GMT

First published on TECHNET on Mar 28, 2016
Hello AskPerf!

Wanted to send you a very long overdue note on the current status of the AskPerf Blog site. We are in a transition period on ownership of this blog site going forward. I personally have moved on to another team, and the remaining Performance folks have as well. With that said, a decision will be made hopefully soon, on the future of this blog.

Thank you as always for your support and active participation in our posts.

-Blake