When you use Service Management Automation (SMA) to create and run runbooks, a couple of key tasks are the testing of runbooks during authoring and the troubleshooting of production runbook jobs. In both cases, you need the runbooks to produce appropriate messages so that you can learn what is going on during execution. Fortunately, PowerShell Workflow, the engine that powers SMA runbooks, has multiple message streams that you can control for optimal benefit during testing and troubleshooting. In this post, I will cover some concepts and best practices for controlling runbook streams that will help you debug and troubleshoot your runbooks.
Testing in SMA
The SMA runbook authoring page is where you create runbooks. It has a pane for editing PowerShell workflow code, a pane for viewing test output, and controls for inserting activities, runbooks, and global settings (see the screenshot below in the Runbook Output for Debugging section). As you author a SMA runbook you will probably test it at key points to assure that the functionality works as expected. Initiating a test for a draft runbook is easy: you simply click the Test button. When the test job is initiated, the Output Pane appears and most streams from the runbook are written into this pane. Of course, the information you put into the streams and the particular streams you use are critical for how well you can debug the runbook. To help you with this, in the next sections I will show you how to control the various streams that PowerShell workflow makes available.
Controlling Runbook Output
PowerShell Workflow supports six different streams, each with a different purpose; these are the Output, Progress, Warning, Error, Verbose, and Debug streams. The Output stream is where output objects from the runbook are written, and the other streams are message streams for particular purposes. Runbooks can write to any of these streams via the cmdlets Write-Output, Write-Progress, Write-Warning, Write-Error, Write-Verbose, and Write-Debug. By learning when to use each stream, you can maximize their usefulness to you during the authoring/debugging phase of runbook creation and during the troubleshooting of production runbook jobs.
In SMA there are two main places where you will want to view runbook streams – these are the runbook authoring page and the runbook job page. In the runbook authoring page you will want to view streams as you debug the runbook during runbook creation and testing. In the runbook job page you will want to view streams for troubleshooting of runbook job issues after the runbook has been published and is running in the production environment.
Runbook Output for Debugging
During runbook authoring when you test a runbook, by default only the Output, Warning, and Error streams are written to the Output Pane. In the screenshot below you can see this behavior.
The reason that only these three streams are written is related to the default preferences set for the streams (see this article on PowerShell preference variables for more information). As you can see in the screenshot below the default preferences for Warning, Error, and Progress are “Continue”, while the preferences for Verbose and Debug are “SilentlyContinue”. This means that when Write-Verbose or Write-Debug cmdlets are used the messages will not be written to the Output Pane (unless the cmdlet calls contain the –Verbose or –Debug switches).
Changing the default settings for testing
During testing you can override the default preferences for the Verbose and Debug streams.
-
To view all Verbose messages in the Output Pane you can set $VerbosePreference=”Continue” at the top of the runbook.
- Or if you only want to see select Verbose messages you can add the “-Verbose” switch to specific Write-Verbose calls (or to any cmdlet with Verbose messages).
-
To view all Debug messages in the Output Pane you can set $DebugPreference=”Continue” at the top of the runbook.
- Or if you only want to see select Debug messages you can add the “-Debug” switch to specific Write-Debug calls (or any cmdlet with Debug messages).
- Important: If you set $DebugPreference=”Continue” in the runbook then you must also set $WarningPreference=”Continue” and $ErrorActionPreference=”Continue” right after it to avoid a terminating exception (unhandled error) being thrown when Write-Warning or Write-Error are called (this is an issue being investigated).
The code below shows these preferences being set in a runbook.
Best Practice: Use Write-Verbose for SMA runbook debugging during authoring. Don’t use Write-Debug or Write-Output for debug messages: Write-Debug is more complicated to use currently, and Write-Output is meant only for output objects that will be used as input to another activity, cmdlet, or runbook.
Best Practice: Use Write-Verbose for general or progress comments you want to appear in the job output for testing and troubleshooting. Again, never use Write-Output for this.
Best Practice: Don’t use Write-Progress in your runbooks. Progress records never appear in the test Output Pane (because they are meant to be displayed in a progress bar), thus they are not useful for debugging. In the next section about production jobs, you will see that there may be some use for Progress records for troubleshooting, but these are only the Progress records automatically generated by the PowerShell workflow engine.
Runbook Output for Production Job Logs
After you have finished authoring your runbook and have published it for use in production, there are a couple of considerations you should keep in mind in relation to logging of the output streams. These considerations are (1) the amount of data stored in the database for each job and (2) the amount of information you need to troubleshoot any possible runbook job issues.
For production jobs (just like test jobs) by default only the Output, Warning, and Error records are written to the database for each job. The reason these records are stored is that they are the most important pieces of information needed on a routine basis. Note that if you set any of the stream preference variables discussed above to ”SilentlyContinue” within a runbook and publish it, then in production execution the runbook will not output any messages for these streams, even if the runbook configuration has logging enabled for these streams.
Below are two screenshots for the Dashboard and History pages of a job which illustrate how the Output, Warning, and Error records appear by default. The Progress, Verbose, and Debug records are not stored by default in order to save database space. (For more information see the blog on Monitoring and Troubleshooting Your Runbooks .)
Changing the default settings for production troubleshooting
You can change the default settings for the logging of the Debug, Verbose, and Progress records in the Configure page for each runbook. See the screenshot below where these settings have been changed to True.
Note that these configuration settings apply only during production runs of the published runbook, and are not obeyed in the runbook authoring/testing page while the runbook is in draft mode. You can also change these settings using the Set-SmaRunbookConfiguration cmdlet.
Important: If you enable Debug logging through the runbook configuration, then you must include $WarningPreference=”Continue” and $ErrorActionPreference=”Continue” in the runbook to avoid a terminating exception (unhandled error) occurring if an Error or Warning are written (issue being investigated).
Best Practice: Use Write-Verbose for general, progress, or debugging information that you want to appear in the job history for troubleshooting production SMA runbooks. Don’t use Write-Debug or Write-Output for this information.
In the screenshot below you can see the result of enabling the logging of the Debug, Verbose, and Progress records.
You can see that much more information is now available in the Job History page. This extra information can be essential for troubleshooting production problems with a runbook. However, this extra information will also be stored in the database and can lead to rapid filling of the database if you have lots of jobs.
Best Practice: Keep the default setting of Off for the Debug, Verbose, and Progress streams, and turn them on only when you need to troubleshoot issues.
The Progress records are especially numerous. By default the workflow engine writes two Progress records for each activity and cmdlet in the workflow – one Progress record when the activity starts and one Progress record when the activity completes. Unless you need this information to track the progress of a runbook for troubleshooting, you should make sure that Progress logging is turned off.
Best Practice: Don’t use Write-Progress in your runbooks. The Progress records that are automatically generated by the PowerShell workflow engine provide plenty of information if you need to track progress during troubleshooting.
Summary
As you can see, it is easy to control the message streams from runbooks for optimal debugging and troubleshooting with minimum filling of the database. Go ahead and play around with this in a scratch runbook and see for yourself how you can control the various streams.