Configuring Application Verifier as a Testing Tool for Windows Vista Compatibility

[Chris Jackson]

First posted to MSDN on Jan, 11 2007

When I speak with customers about compatibility testing in preparation for an upgrade to Windows Vista, one of the tools I always introduce is Application Verifier.

Application Verifier is a great tool for developers. It runs a number of runtime checks to see if you are doing anything that may comprimise the reliability or compatibility of your application. (Consequently, as a runtime analysis tool, it depends on you having a thorough test plan to ensure adequate code coverage.) When AppVerifier runs across a noteworthy issue, it throws a stop error, which breaks you into a debugger right at that line of code, which is extremely useful for a developer.

However, throwing a stop error is significantly less useful for a tester who is not familiar with, and may not have access to, the code. If you don't have a debugger attached, that error will frequently just bubble up, never caught, and eventually lead to a crash of the application. Not particularly helpful!

Of course, you can configure Appliation Verifier not to break into the debugger, but the user experience there is not optimized for configuring in bulk. If you start up Application Verifier, and then click on one of the test types, you can go to Edit -> Verifier Stop Options. You then see a list of all of the verifier stops, and whether or not you will throw an error. However, the first thing you will note is that you can't multi-select the Verifier Stop codes. You have to go one by one. Second, if you select one, configure it, and then move to the next one - try going back. You'll find that the configuration change you just made has gone away. You have to press OK after configuring each and every stop error and then re-open the dialog box to configure the next one. Some of these tests have quite a few stop codes, so that can be quite time consuming!

There are a couple of ways to simplify that process. If you have already done it once, all you need to do is grab the registry key that was created (under Image File Execution Options) and export them. Change the name for a new application, and voila - the new application is configured. However, if you haven't configured it once already, the description above isn't likely to sound particularly tempting.

Fortunately, Application Verifier has a command line interface, so you can script that process. Even more fortunately, only one person has to write that into a little command fine - and I figured that person may as well be me. You can use the following script to enable and configure a subset of Application Verifier tests that will be interesting in terms of collecting data to support compatibility and reliability testing. (You can always add or remove based on your needs.)

Just copy this text and paste it into Notepad, and save with a .cmd file extension. I call mine AppVerifier Test Config.cmd.

@echo off
if (%1)==() goto NoArgument
@echo on
appverif.exe -enable Exceptions Handles Heaps Locks -for %1
appverif.exe -configure 0x0650 -for %1 -with ErrorReport=0
appverif.exe -enable Handles -for %1
appverif.exe -configure 0x0300 0x0301 0x0302 0x0303 0x0304 0x0305 -for %1 -with ErrorReport=0
appverif.exe -enable Heaps -for %1
appverif.exe -configure 0x0001 0x0002 0x0003 0x0004 0x0005 0x0006 0x0007 0x0008 0x0009 0x000A 0x000B 0x000C 0x000D 0x000E 0x000F 0x0010 0x0011 0x0012 0x0013 0x0014 -for %1 -with ErrorReport=0
appverif.exe -enable Locks -for %1
appverif.exe -configure 0x0201 0x0202 0x0203 0x0204 0x0205 0x0206 0x0207 0x0208 0x0209 0x0210 0x0211 0x0212 0x0213 0x0214 0x0215 -for %1 -with ErrorReport=0
appverif.exe -enable Memory -for %1
appverif.exe -configure 0x0601 0x0602 0x0603 0x0604 0x0605 0x0606 0x0607 0x0608 0x0609 0x060A 0x060B 0x060C 0x060D 0x060E 0x060F 0x0610 0x0612 0x0613 0x0614 0x0615 0x0616 0x0617 0x0618 0x0619 0x061A 0x061B 0x061C 0x061D 0x061E -for %1 -with ErrorReport=0
appverif.exe -enable Threadpool -for %1
appverif.exe -configure 0x0700 0x0701 0x0702 0x0703 0x0704 0x0705 0x0706 0x0707 0x0708 -for %1 -with ErrorReport=0
appverif.exe -enable TLS -for %1
appverif.exe -configure 0x0350 0x0351 0x0352 -for %1 -with ErrorReport=0
appverif.exe -enable FilePaths -for %1
appverif.exe -configure 0x2400 0x2401 0x2402 0x2404 0x2405 0x2406 0x2407 0x2408 0x240A -for %1 -with ErrorReport=0
appverif.exe -enable HighVersionLie -for %1
appverif.exe -configure 0x2200 0x2201 0x2202 0x2203 0x2204 -for %1 -with ErrorReport=0
appverif.exe -enable InteractiveServices -for %1
appverif.exe -configure 0x2800 0x2801 0x2802 0x2803 0x2804 0x2805 0x2806 0x2807 -for %1 -with ErrorReport=0
appverif.exe -enable KernelModeDriverInstall -for %1
appverif.exe -configure 0x2305 0x2306 0x2307 0x2308 0x2309 0x230A 0x230B 0x230C 0x238D -for %1 -with ErrorReport=0
appverif.exe -enable LuaPriv -for %1
appverif.exe -configure 0x3300 0x3301 0x3302 0x3303 0x3304 0x3305 0x3306 0x3307 0x3308 0x3309 0x330A 0x330B 0x330C 0x330D 0x330E 0x330F 0x3310 0x3311 0x3312 0x3313 0x3314 0x3315 0x3316 0x3317 0x3318 0x3319 0x331A 0x331B 0x331C 0x331D 0x331E 0x331F 0x3320 0x3321 0x3322 0x3323 0x3324 0x3325 0x3326 0x3327 0x3328 0x3329 0x332A 0x332B 0x332C 0x332D 0x332E 0x332F 0x3330 0x3331 0x3332 0x3333 0x3334 0x3335 0x3336 0x3337 0x3338 -for %1 -with ErrorReport=0
appverif.exe -enable DangerousAPIs -for %1
appverif.exe -configure 0x0100 0x0101 0x0102 0x0103 0x0104 -for %1 -with ErrorReport=0
appverif.exe -enable DirtyStacks -for %1
goto End
:NoArgument
echo Please specify the application name (without path)
:End

Now, I just need to run AppVerifier Test Config "My Application Name Without Path". Much easier than going through all of that clicking!

Keep in mind that Application Verifier is a "set and forget" tool. You will be monitoring that application every single time that you run it from now until the end of time (or until you go into Application Verifier and remove it). When you want to see the data that you collected, just launch the Application Verifier GUI, from the View menu select Logs, and then select the test run that you want to review data for. You can then either press View to open the data in your default XML editor, or you can press the Save As button and save it somewhere as XML, and then use another tool to open it up and review. Personally, I prefer to save it and open that XML inside of Microsoft Office Excel.

When you are done testing that application, you can right click on it and select Delete Application to stop monitoring. Of course, you can script this as well, but since it's only a couple of mouse clicks away I typically end up going through the UI.

The output from Application Verifier is quite raw. It has a lot of data. Standard User Analyzer uses Application Verifier, and it performs a lot of work on the output of LuaPriv to make it easier to understand and to understand how you can mitigate the issues that it detected. We are not so lucky with the rest. However, while it is a fairly big step down from many of the other tools included in the Application Compatibility Toolkit 5.0, it is not nearly as big of a step down as going all of the way to a debugger. The less time you spend in a debugger, the more productive you will be. Hopefully this technique can help you track down more quality and compatibility issues without resorting to scrolling through page after page of symbol-free assembler for somebody else's code!