DTrace on Windows

[Hari_Pulapaka]

Here at Microsoft, we are always looking to engage with open source communities to produce better solutions for the community and our customers . One of the more useful debugging advances that have arrived in the last decade is DTrace. DTrace of course needs no introduction: it’s a dynamic tracing framework that allows an admin or developer to get a real-time look into a system either in user or kernel mode. DTrace has a C-style high level and powerful programming language that allows you to dynamically insert trace points. Using these dynamically inserted trace points, you can filter on conditions or errors, write code to analyze lock patterns, detect deadlocks, etc. ETW while powerful, is static and does not provide the ability to programmatically insert trace points at runtime.  

There are a lot of websites and resources from the community to learn about DTrace. One of the most comprehensive one is the Dynamic Tracing Guide html book available on dtrace.org website. This ebook describes DTrace in detail and is the authoritative guide for DTrace. We also have Windows specific examples below which will provide more info.

Starting in 2016, the OpenDTrace effort began on GitHub that  tried to ensure a portable implementation of DTrace for different operating systems. We decided to add support for DTrace on Windows using this OpenDTrace port.

We have created a Windows branch for “DTrace on Windows” under the OpenDTrace project on GitHub. All our changes made to support DTrace on Windows are available here. Over the next few months, we plan to work with the OpenDTrace community to merge our changes. All our source code is also available at the 3^rd party sources website maintained by Microsoft.   

Without further ado, let’s get into how to setup and use DTrace on Windows.

Install and Run DTrace

Prerequisites for using the feature

• Windows 10 insider build 18342 or higher
• Only available on x64 Windows and captures tracing info only for 64-bit processes
• Windows Insider Program is enabled and configured with valid Windows Insider Account
  • Visit Settings->Update & Security->Windows Insider Program for details

Instructions:

1. BCD configuration set:
   1. bcdedit /set dtrace on
   2. Note, you need to set the bcdedit option again, if you upgrade to a new Insider build
2. Download and install the DTrace package from download center.
   1. This installs the user mode components, drivers and additional feature on demand packages necessary for DTrace to be functional.
3. Optional: Update the PATH environment variable to include C:\Program Files\DTrace
   1. set PATH=%PATH%;"C:\Program Files\DTrace"
4. Setup symbol path
   1. Create a new directory for caching symbols locally. Example: mkdir c:\symbols
   2. Set _NT_SYMBOL_PATH=srv*C:\symbols*https://msdl.microsoft.com/download/symbols
   3. DTrace automatically downloads the symbols necessary from the symbol server and caches to the local path.
5. Optional: Setup Kernel debugger connection to the target machine (MSDN link). This is only required if you want to trace Kernel events using FBT or other providers.
   1. Note that you will need to disable Secureboot and Bitlocker on C:, (if enabled), if you want to setup a kernel debugger. 
6. Reboot target machine

 

Running DTrace

Launch CMD prompt in administrator mode

Get started with sample one-liners:

# Syscall summary by program for 5 seconds: 
dtrace -Fn "tick-5sec { exit(0);} syscall:::entry{ @num[pid,execname] = count();} "
 
# Summarize timer set/cancel program for 3 seconds: 
dtrace -Fn "tick-3sec { exit(0);} syscall::Nt*Timer*:entry { @[probefunc, execname, pid] = count();}"
 
# Dump System Process kernel structure: (requires symbol path to be set)
dtrace -n "BEGIN{print(*(struct nt`_EPROCESS *) nt`PsInitialSystemProcess);exit(0);}"
 
# Tracing paths through NTFS when running notepad.exe (requires KD attach): Run below command and launch notepad.exe
dtrace -Fn "fbt:ntfs::/execname==\"notepad.exe\"/{}"

The command dtrace -lvn syscall::: will list all the probes and their parameters available from the syscall provider.

The following are some of the providers available on Windows and what they instrument.

• syscall – NTOS system calls
• fbt (Function Boundary Tracing) – Kernel function entry and returns
• pid – User-mode process tracing. Like kernel-mode FBT, but also allowing the instrumentation of arbitrary function offsets.
• etw (Event Tracing for Windows) – Allows probes to be defined for ETW This provider helps to leverage existing operating system instrumentation in DTrace.
  • This is one addition we have done to DTrace to allow it to expose and gain all the information that Windows already provides in ETW.

We have more Windows sample scripts applicable for Windows scenarios in the samples directory of the source.

How to file feedback?

DTrace on Windows is very different from our typical features on Windows and we are going to rely on our Insider community to guide us. If you hit any problems or bugs, please use Feedback hub to let us know.

1. Launch feedback hub by clicking this link
2. Select Add new feedback.
3. Please provide a detailed description of the issue or suggestion.
   1. Currently, we do not automatically collect any debug traces, so your verbatim feedback is crucial for understanding and reproducing the issue. Pass on any verbose logs.
   2. You can set DTRACE_DEBUG environment variable to 1 to collect verbose dtrace logs.
4. Submit

DTrace Architecture

Let’s talk a little about the internals and architecture of how we supported DTrace. As mentioned, DTrace on Windows is a port of OpenDTrace and reuses much of its user mode components and architecture. Users interact with DTrace through the dtrace command, which is a generic front-end to the DTrace engine. D scripts get compiled to an intermediate format (DIF) in user-space and sent to the DTrace kernel component for execution, sometimes called as the DIF Virtual Machine. This runs in the dtrace.sys driver.

Traceext.sys (trace extension) is a new kernel extension driver we added, which allows Windows to expose functionality that DTrace relies on to provide tracing. The Windows kernel provides callouts during stackwalk or memory accesses which are then implemented by the trace extension.

All APIs and functionality used by dtrace.sys are documented calls.

Security

Security of Windows is key for our customers and the security model of DTrace makes it ideally suited to Windows. The DTrace guide, linked above talks about DTrace security and performance impact. It would be useful for anyone interested in this space to read that section. At a high level, DTrace uses an intermediate form which is validated for safety and runs in its own execution environment (think C# or Java). This execution environment also handles any run time errors to avoid crashing the system. In addition, the cost of having a probe is minimal and should not visibly affect the system performance unless you enable too many probes in performance sensitive paths.

DTrace on Windows also leverages the Windows security model in useful ways to enhance its security for our customers.

1. To connect to the DTrace trace engine, your account needs to be part of the admin or LocalSystem group
2. Events originating from kernel mode (FBT, syscalls with ‘kernel’ previous mode, etc.), are only traceable if Kernel debugger is attached
3. To read kernel-mode memory (probe parameters for kernel-mode originated events, kernel-mode global variables, etc.), the following must be true:
   1. DTrace session security context has either TCB or LoadDriver privilege enabled.
   2. Secure Boot is not active.
4. To trace a user-mode process, the user needs to have:
   1. Debug privilege
   2. DEBUG access to the target process.

Script signing

In addition, we have also updated DTrace on Windows to support signing of d scripts. We follow the same model as PowerShell to support signing of scripts.

There is a system wide DTrace script signing policy knob which controls whether to check for signing or not for DTrace scripts. This policy knob is controlled by the Registry.

By default, we do NOT check for signature on DTrace scripts.

Use the following registry keys to enforce policy at machine or user level.

• User Scope: HKCU\Software\OpenDTrace\Dtrace, ExecutionPolicy, REG_SZ
• Machine Scope: HKLM\Software\OpenDTrace\Dtrace, ExecutionPolicy, REG_SZ

Policy Values:

DTrace policy take the following values.

• “Bypass": do not perform signature checks. This is the default policy. Only set the registry key if you want to deviate from this policy.
• "Unrestricted": Do not perform checks on local files, allow user's consent to use unsigned remote files.
• "RemoteSigned": Do not perform checks on local files, requires a valid and trusted signature for remote files.
• "AllSigned": Require valid and trusted signature for all files.
• "Restricted": Script file must be installed as a system component and have a signature from the trusted source.

You can also set policy by defining the environment variable DTRACE_EXECUTION_POLICY to the required value.

Conclusion

We are very excited to release the first version of DTrace on Windows. We look forward to feedback from the Windows Insider community.

Cheers,

DTrace Team (Andrey Shedel, Gopikrishna Kannan, & Hari Pulapaka)